pwc-archive/papers-with-abstracts huggingface dataset:
Understanding Membership Inferences on Well-Generalized Learning Models
Authors: Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiao-Feng Wang, Haixu Tang, Carl A. Gunter, Kai Chen
Date: 2018-02-13 00:00:00
Tasks: BIG-bench Machine Learning, Inference Attack, Membership Inference Attack

Membership Inference Attack (MIA) determines the presence of a record in a
machine learning model's training data by querying the model. Prior work has
shown that the attack is feasible when the model is overfitted to its training
data or when the adversary controls the training algorithm. However, when the
model is not overfitted and the adversary does not control the training
algorithm, the threat is not well understood. In this paper, we report a study
that discovers overfitting to be a sufficient but not a necessary condition for
an MIA to succeed. More specifically, we demonstrate that even a
well-generalized model contains vulnerable instances subject to a new
generalized MIA (GMIA). In GMIA, we use novel techniques for selecting
vulnerable instances and detecting their subtle influences ignored by
overfitting metrics. Specifically, we successfully identify individual records
with high precision in real-world datasets by querying black-box machine
learning models. Further we show that a vulnerable record can even be
indirectly attacked by querying other related records and existing
generalization techniques are found to be less effective in protecting the
vulnerable instances. Our findings sharpen the understanding of the fundamental
cause of the problem: the unique influences the training instance may have on
the model.
Probabilistic Warnings in National Security Crises: Pearl Harbor Revisited
Authors: David M. Blum, M. Elisabeth Pate-Cornell
Date: 2018-02-13 00:00:00

Imagine a situation where a group of adversaries is preparing an attack on
the United States or U.S. interests. An intelligence analyst has observed some
signals, but the situation is rapidly changing. The analyst faces the decision
to alert a principal decision maker that an attack is imminent, or to wait
until more is known about the situation. This warning decision is based on the
analyst's observation and evaluation of signals, independent or correlated, and
on her updating of the prior probabilities of possible scenarios and their
outcomes. The warning decision also depends on the analyst's assessment of the
crisis' dynamics and perception of the preferences of the principal decision
maker, as well as the lead time needed for an appropriate response. This
article presents a model to support this analyst's dynamic warning decision. As
with most problems involving warning, the key is to manage the tradeoffs
between false positives and false negatives given the probabilities and the
consequences of intelligence failures of both types. The model is illustrated
by revisiting the case of the attack on Pearl Harbor in December 1941. It shows
that the radio silence of the Japanese fleet carried considerable information
(Sir Arthur Conan Doyle's "dog in the night" problem), which was misinterpreted
at the time. Even though the probabilities of different attacks were relatively
low, their consequences were such that the Bayesian dynamic reasoning described
here may have provided valuable information to key decision makers.
Learning a SAT Solver from Single-Bit Supervision
Authors: Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, David L. Dill
Date: 2018-02-11 00:00:00
Proceeding: ICLR 2019 5

We present NeuroSAT, a message passing neural network that learns to solve
SAT problems after only being trained as a classifier to predict
satisfiability. Although it is not competitive with state-of-the-art SAT
solvers, NeuroSAT can solve problems that are substantially larger and more
difficult than it ever saw during training by simply running for more
iterations. Moreover, NeuroSAT generalizes to novel distributions; after
training only on random SAT problems, at test time it can solve SAT problems
encoding graph coloring, clique detection, dominating set, and vertex cover
problems, all on a range of distributions over small random graphs.
Satellite Image Forgery Detection and Localization Using GAN and One-Class Classifier
Authors: Sri Kalyan Yarlagadda, David Güera, Paolo Bestagini, Fengqing Maggie Zhu, Stefano Tubaro, Edward J. Delp
Date: 2018-02-13 00:00:00
Tasks: Generative Adversarial Network, Image Forgery Detection, One-class classifier

Current satellite imaging technology enables shooting high-resolution
pictures of the ground. As any other kind of digital images, overhead pictures
can also be easily forged. However, common image forensic techniques are often
developed for consumer camera images, which strongly differ in their nature
from satellite ones (e.g., compression schemes, post-processing, sensors,
etc.). Therefore, many accurate state-of-the-art forensic algorithms are bound
to fail if blindly applied to overhead image analysis. Development of novel
forensic tools for satellite images is paramount to assess their authenticity
and integrity. In this paper, we propose an algorithm for satellite image
forgery detection and localization. Specifically, we consider the scenario in
which pixels within a region of a satellite image are replaced to add or remove
an object from the scene. Our algorithm works under the assumption that no
forged images are available for training. Using a generative adversarial
network (GAN), we learn a feature representation of pristine satellite images.
A one-class support vector machine (SVM) is trained on these features to
determine their distribution. Finally, image forgeries are detected as
anomalies. The proposed algorithm is validated against different kinds of
satellite images containing forgeries of different size and shape.
Learning via social awareness: Improving a deep generative sketching model with facial feedback
Authors: Natasha Jaques, Jennifer McCleary, Jesse Engel, David Ha, Fred Bertsch, Rosalind Picard, Douglas Eck
Date: 2018-02-13 00:00:00
Tasks: AI Agent

In the quest towards general artificial intelligence (AI), researchers have
explored developing loss functions that act as intrinsic motivators in the
absence of external rewards. This paper argues that such research has
overlooked an important and useful intrinsic motivator: social interaction. We
posit that making an AI agent aware of implicit social feedback from humans can
allow for faster learning of more generalizable and useful representations, and
could potentially impact AI safety. We collect social feedback in the form of
facial expression reactions to samples from Sketch RNN, an LSTM-based
variational autoencoder (VAE) designed to produce sketch drawings. We use a
Latent Constraints GAN (LC-GAN) to learn from the facial feedback of a small
group of viewers, by optimizing the model to produce sketches that it predicts
will lead to more positive facial expressions. We show in multiple independent
evaluations that the model trained with facial feedback produced sketches that
are more highly rated, and induce significantly more positive facial
expressions. Thus, we establish that implicit social feedback can improve the
output of a deep learning model.
HiGrad: Uncertainty Quantification for Online Learning and Stochastic Approximation
Authors: Weijie J. Su, Yuancheng Zhu
Date: 2018-02-13 00:00:00
Tasks: Uncertainty Quantification

Stochastic gradient descent (SGD) is an immensely popular approach for online learning in settings where data arrives in a stream or data sizes are very large. However, despite an ever-increasing volume of work on SGD, much less is known about the statistical inferential properties of SGD-based predictions. Taking a fully inferential viewpoint, this paper introduces a novel procedure termed HiGrad to conduct statistical inference for online learning, without incurring additional computational cost compared with SGD. The HiGrad procedure begins by performing SGD updates for a while and then splits the single thread into several threads, and this procedure hierarchically operates in this fashion along each thread. With predictions provided by multiple threads in place, a $t$-based confidence interval is constructed by decorrelating predictions using covariance structures given by a Donsker-style extension of the Ruppert--Polyak averaging scheme, which is a technical contribution of independent interest. Under certain regularity conditions, the HiGrad confidence interval is shown to attain asymptotically exact coverage probability. Finally, the performance of HiGrad is evaluated through extensive simulation studies and a real data example. An R package \texttt{higrad} has been developed to implement the method.
GILBO: One Metric to Measure Them All
Authors: Alexander A. Alemi, Ian Fischer
Date: 2018-02-13 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: All

We propose a simple, tractable lower bound on the mutual information
contained in the joint generative density of any latent variable generative
model: the GILBO (Generative Information Lower BOund). It offers a
data-independent measure of the complexity of the learned latent variable
description, giving the log of the effective description length. It is
well-defined for both VAEs and GANs. We compute the GILBO for 800 GANs and VAEs
each trained on four datasets (MNIST, FashionMNIST, CIFAR-10 and CelebA) and
discuss the results.
SimplE Embedding for Link Prediction in Knowledge Graphs
Authors: Seyed Mehran Kazemi, David Poole
Date: 2018-02-13 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Knowledge Graphs, Link Prediction, Prediction

Knowledge graphs contain knowledge about the world and provide a structured
representation of this knowledge. Current knowledge graphs contain only a small
subset of what is true in the world. Link prediction approaches aim at
predicting new links for a knowledge graph given the existing links among the
entities. Tensor factorization approaches have proved promising for such link
prediction problems. Proposed in 1927, Canonical Polyadic (CP) decomposition is
among the first tensor factorization approaches. CP generally performs poorly
for link prediction as it learns two independent embedding vectors for each
entity, whereas they are really tied. We present a simple enhancement of CP
(which we call SimplE) to allow the two embeddings of each entity to be learned
dependently. The complexity of SimplE grows linearly with the size of
embeddings. The embeddings learned through SimplE are interpretable, and
certain types of background knowledge can be incorporated into these embeddings
through weight tying. We prove SimplE is fully expressive and derive a bound on
the size of its embeddings for full expressivity. We show empirically that,
despite its simplicity, SimplE outperforms several state-of-the-art tensor
factorization techniques. SimplE's code is available on GitHub at
https://github.com/Mehran-k/SimplE.
Learning Confidence for Out-of-Distribution Detection in Neural Networks
Authors: Terrance DeVries, Graham W. Taylor
Date: 2018-02-13 00:00:00
Tasks: Out-of-Distribution Detection

Modern neural networks are very powerful predictive models, but they are
often incapable of recognizing when their predictions may be wrong. Closely
related to this is the task of out-of-distribution detection, where a network
must determine whether or not an input is outside of the set on which it is
expected to safely perform. To jointly address these issues, we propose a
method of learning confidence estimates for neural networks that is simple to
implement and produces intuitively interpretable outputs. We demonstrate that
on the task of out-of-distribution detection, our technique surpasses recently
proposed techniques which construct confidence based on the network's output
distribution, without requiring any additional labels or access to
out-of-distribution examples. Additionally, we address the problem of
calibrating out-of-distribution detectors, where we demonstrate that
misclassified in-distribution examples can be used as a proxy for
out-of-distribution examples.
Making "fetch" happen: The influence of social and linguistic context on nonstandard word growth and decline
Authors: Ian Stewart, Jacob Eisenstein
Date: 2018-02-09 00:00:00
Proceeding: EMNLP 2018 10

In an online community, new words come and go: today's "haha" may be replaced
by tomorrow's "lol." Changes in online writing are usually studied as a social
process, with innovations diffusing through a network of individuals in a
speech community. But unlike other types of innovation, language change is
shaped and constrained by the system in which it takes part. To investigate the
links between social and structural factors in language change, we undertake a
large-scale analysis of nonstandard word growth in the online community Reddit.
We find that dissemination across many linguistic contexts is a sign of growth:
words that appear in more linguistic contexts grow faster and survive longer.
We also find that social dissemination likely plays a less important role in
explaining word growth and decline than previously hypothesized.
A theoretical guideline for designing an effective adaptive particle swarm
Authors: Mohammad Reza Bonyadi
Date: 2018-02-13 00:00:00

In this paper we theoretically investigate underlying assumptions that have
been used for designing adaptive particle swarm optimization algorithms in the
past years. We relate these assumptions to the movement patterns of particles
controlled by coefficient values (inertia weight and acceleration coefficient)
and introduce three factors, namely the autocorrelation of the particle
positions, the average movement distance of the particle in each iteration, and
the focus of the search, that describe these movement patterns. We show how
these factors represent movement patterns of a particle within a swarm and how
they are affected by particle coefficients (i.e., inertia weight and
acceleration coefficients). We derive equations that provide exact coefficient
values to guarantee achieving a desired movement pattern defined by these three
factors within a swarm. We then relate these movements to the searching
capability of particles and provide guideline for designing potentially
successful adaptive methods to control coefficients in particle swarm. Finally,
we propose a new simple time adaptive particle swarm and compare its results
with previous adaptive particle swarm approaches. Our experiments show that the
theoretical findings indeed provide a beneficial guideline for successful
adaptation of the coefficients in the particle swarm optimization algorithm.
Fixing a Broken ELBO
Authors: Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, Kevin Murphy
Date: 2017-11-01 00:00:00
Proceeding: ICML 2018 7
Tasks: Representation Learning

Recent work in unsupervised representation learning has focused on learning
deep directed latent-variable models. Fitting these models by maximizing the
marginal likelihood or evidence is typically intractable, thus a common
approximation is to maximize the evidence lower bound (ELBO) instead. However,
maximum likelihood training (whether exact or approximate) does not necessarily
result in a good latent representation, as we demonstrate both theoretically
and empirically. In particular, we derive variational lower and upper bounds on
the mutual information between the input and the latent variable, and use these
bounds to derive a rate-distortion curve that characterizes the tradeoff
between compression and reconstruction accuracy. Using this framework, we
demonstrate that there is a family of models with identical ELBO, but different
quantitative and qualitative characteristics. Our framework also suggests a
simple new method to ensure that latent variable models with powerful
stochastic decoders do not ignore their latent code.
Clustering and Semi-Supervised Classification for Clickstream Data via Mixture Models
Authors: Michael P. B. Gallaugher, Paul D. McNicholas
Date: 2018-02-13 00:00:00
Tasks: Clustering, General Classification

Finite mixture models have been used for unsupervised learning for some time, and their use within the semi-supervised paradigm is becoming more commonplace. Clickstream data is one of the various emerging data types that demands particular attention because there is a notable paucity of statistical learning approaches currently available. A mixture of first-order continuous time Markov models is introduced for unsupervised and semi-supervised learning of clickstream data. This approach assumes continuous time, which distinguishes it from existing mixture model-based approaches; practically, this allows account to be taken of the amount of time each user spends on each webpage. The approach is evaluated, and compared to the discrete time approach, using simulated and real data.
Network Estimation from Point Process Data
Authors: Benjamin Mark, Garvesh Raskutti, Rebecca Willett
Date: 2018-02-13 00:00:00
Tasks: Point Processes

Consider observing a collection of discrete events within a network that
reflect how network nodes influence one another. Such data are common in spike
trains recorded from biological neural networks, interactions within a social
network, and a variety of other settings. Data of this form may be modeled as
self-exciting point processes, in which the likelihood of future events depends
on the past events. This paper addresses the problem of estimating
self-excitation parameters and inferring the underlying functional network
structure from self-exciting point process data. Past work in this area was
limited by strong assumptions which are addressed by the novel approach here.
Specifically, in this paper we (1) incorporate saturation in a point process
model which both ensures stability and models non-linear thresholding effects;
(2) impose general low-dimensional structural assumptions that include
sparsity, group sparsity and low-rankness that allows bounds to be developed in
the high-dimensional setting; and (3) incorporate long-range memory effects
through moving average and higher-order auto-regressive components. Using our
general framework, we provide a number of novel theoretical guarantees for
high-dimensional self-exciting point processes that reflect the role played by
the underlying network structure and long-term memory. We also provide
simulations and real data examples to support our methodology and main results.
Challenging Images For Minds and Machines
Authors: Amir Rosenfeld, John K. Tsotsos
Date: 2018-02-13 00:00:00
Tasks: BIG-bench Machine Learning, Position

There is no denying the tremendous leap in the performance of machine
learning methods in the past half-decade. Some might even say that specific
sub-fields in pattern recognition, such as machine-vision, are as good as
solved, reaching human and super-human levels. Arguably, lack of training data
and computation power are all that stand between us and solving the remaining
ones. In this position paper we underline cases in vision which are challenging
to machines and even to human observers. This is to show limitations of
contemporary models that are hard to ameliorate by following the current trend
to increase training data, network capacity or computational power. Moreover,
we claim that attempting to do so is in principle a suboptimal approach. We
provide a taster of such examples in hope to encourage and challenge the
machine learning community to develop new directions to solve the said
difficulties.
Learning Filter Scale and Orientation In CNNs
Authors: Ilker Cam, F. Boray Tek
Date: 2018-02-13 00:00:00

Convolutional neural networks have many hyperparameters such as the filter
size, number of filters, and pooling size, which require manual tuning. Though
deep stacked structures are able to create multi-scale and hierarchical
representations, manually fixed filter sizes limit the scale of representations
that can be learned in a single convolutional layer.
  This paper introduces a new adaptive filter model that allows variable scale
and orientation. The scale and orientation parameters of filters can be learned
using back propagation. Therefore, in a single convolution layer, we can create
filters of different scale and orientation that can adapt to small or large
features and objects. The proposed model uses a relatively large base size
(grid) for filters. In the grid, a differentiable function acts as an envelope
for the filters. The envelope function guides effective filter scale and
shape/orientation by masking the filter weights before the convolution.
Therefore, only the weights in the envelope are updated during training.
  In this work, we employed a multivariate (2D) Gaussian as the envelope
function and showed that it can grow, shrink, or rotate by updating its
covariance matrix during back propagation training . We tested the new filter
model on MNIST, MNIST-cluttered, and CIFAR-10 and compared the results with the
networks that used conventional convolution layers. The results demonstrate
that the new model can effectively learn and produce filters of different
scales and orientations in a single layer. Moreover, the experiments show that
the adaptive convolution layers perform equally; or better, especially when
data includes objects of varying scale and noisy backgrounds.
Incremental Learning Through Deep Adaptation
Authors: Amir Rosenfeld, John K. Tsotsos
Date: 2017-05-11 00:00:00
Proceeding: ICLR 2018 1
Tasks: Continual Learning, image-classification, Image Classification, Incremental Learning, Quantization

Given an existing trained neural network, it is often desirable to learn new
capabilities without hindering performance of those already learned. Existing
approaches either learn sub-optimal solutions, require joint training, or incur
a substantial increment in the number of parameters for each added domain,
typically as many as the original network. We propose a method called
\emph{Deep Adaptation Networks} (DAN) that constrains newly learned filters to
be linear combinations of existing ones. DANs precisely preserve performance on
the original domain, require a fraction (typically 13\%, dependent on network
architecture) of the number of parameters compared to standard fine-tuning
procedures and converge in less cycles of training to a comparable or better
level of performance. When coupled with standard network quantization
techniques, we further reduce the parameter cost to around 3\% of the original
with negligible or no loss in accuracy. The learned architecture can be
controlled to switch between various learned representations, enabling a single
network to solve a task from multiple different domains. We conduct extensive
experiments showing the effectiveness of our method on a range of image
classification tasks and explore different aspects of its behavior.