pwc-archive/papers-with-abstracts huggingface dataset:
Multi-Generator Generative Adversarial Nets
Authors: Quan Hoang, Tu Dinh Nguyen, Trung Le, Dinh Phung
Date: 2017-08-08 00:00:00

We propose a new approach to train the Generative Adversarial Nets (GANs)
with a mixture of generators to overcome the mode collapsing problem. The main
intuition is to employ multiple generators, instead of using a single one as in
the original GAN. The idea is simple, yet proven to be extremely effective at
covering diverse data modes, easily overcoming the mode collapse and delivering
state-of-the-art results. A minimax formulation is able to establish among a
classifier, a discriminator, and a set of generators in a similar spirit with
GAN. Generators create samples that are intended to come from the same
distribution as the training data, whilst the discriminator determines whether
samples are true data or generated by generators, and the classifier specifies
which generator a sample comes from. The distinguishing feature is that
internal samples are created from multiple generators, and then one of them
will be randomly selected as final output similar to the mechanism of a
probabilistic mixture model. We term our method Mixture GAN (MGAN). We develop
theoretical analysis to prove that, at the equilibrium, the Jensen-Shannon
divergence (JSD) between the mixture of generators' distributions and the
empirical data distribution is minimal, whilst the JSD among generators'
distributions is maximal, hence effectively avoiding the mode collapse. By
utilizing parameter sharing, our proposed model adds minimal computational cost
to the standard GAN, and thus can also efficiently scale to large-scale
datasets. We conduct extensive experiments on synthetic 2D data and natural
image databases (CIFAR-10, STL-10 and ImageNet) to demonstrate the superior
performance of our MGAN in achieving state-of-the-art Inception scores over
latest baselines, generating diverse and appealing recognizable objects at
different resolutions, and specializing in capturing different types of objects
by generators.
Automated Design using Neural Networks and Gradient Descent
Authors: Oliver Hennigh
Date: 2017-10-27 00:00:00
Proceeding: ICLR 2018 1

We propose a novel method that makes use of deep neural networks and gradient
decent to perform automated design on complex real world engineering tasks. Our
approach works by training a neural network to mimic the fitness function of a
design optimization task and then, using the differential nature of the neural
network, perform gradient decent to maximize the fitness. We demonstrate this
methods effectiveness by designing an optimized heat sink and both 2D and 3D
airfoils that maximize the lift drag ratio under steady state flow conditions.
We highlight that our method has two distinct benefits over other automated
design approaches. First, evaluating the neural networks prediction of fitness
can be orders of magnitude faster then simulating the system of interest.
Second, using gradient decent allows the design space to be searched much more
efficiently then other gradient free methods. These two strengths work together
to overcome some of the current shortcomings of automated design.
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
Authors: Li Yi, Lin Shao, Manolis Savva, Haibin Huang, Yang Zhou, Qirui Wang, Benjamin Graham, Martin Engelcke, Roman Klokov, Victor Lempitsky, Yuan Gan, Pengyu Wang, Kun Liu, Fenggen Yu, Panpan Shui, Bingyang Hu, Yan Zhang, Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Minki Jeong, Jaehoon Choi, Changick Kim, Angom Geetchandra, Narasimha Murthy, Bhargava Ramu, Bharadwaj Manda, M. Ramanathan, Gautam Kumar, P Preetham, Siddharth Srivastava, Swati Bhugra, Brejesh lall, Christian Haene, Shubham Tulsiani, Jitendra Malik, Jared Lafer, Ramsey Jones, Siyuan Li, Jie Lu, Shi Jin, Jingyi Yu, Qi-Xing Huang, Evangelos Kalogerakis, Silvio Savarese, Pat Hanrahan, Thomas Funkhouser, Hao Su, Leonidas Guibas
Date: 2017-10-17 00:00:00
Tasks: 3D Part Segmentation, 3D Reconstruction, 3D Shape Reconstruction

We introduce a large-scale 3D shape understanding benchmark using data and
annotation from ShapeNet 3D object database. The benchmark consists of two
tasks: part-level segmentation of 3D shapes and 3D reconstruction from single
view images. Ten teams have participated in the challenge and the best
performing teams have outperformed state-of-the-art approaches on both tasks. A
few novel deep learning architectures have been proposed on various 3D
representations on both tasks. We report the techniques used by each team and
the corresponding performances. In addition, we summarize the major discoveries
from the reported results and possible trends for the future work in the field.
Multiple Source Domain Adaptation with Adversarial Training of Neural Networks
Authors: Han Zhao, Shanghang Zhang, Guanhang Wu, João P. Costeira, José M. F. Moura, Geoffrey J. Gordon
Date: 2017-05-26 00:00:00
Tasks: Domain Adaptation, Sentiment Analysis

While domain adaptation has been actively researched in recent years, most
theoretical results and algorithms focus on the single-source-single-target
adaptation setting. Naive application of such algorithms on multiple source
domain adaptation problem may lead to suboptimal solutions. As a step toward
bridging the gap, we propose a new generalization bound for domain adaptation
when there are multiple source domains with labeled instances and one target
domain with unlabeled instances. Compared with existing bounds, the new bound
does not require expert knowledge about the target distribution, nor the
optimal combination rule for multisource domains. Interestingly, our theory
also leads to an efficient learning strategy using adversarial neural networks:
we show how to interpret it as learning feature representations that are
invariant to the multiple domain shifts while still being discriminative for
the learning task. To this end, we propose two models, both of which we call
multisource domain adversarial networks (MDANs): the first model optimizes
directly our bound, while the second model is a smoothed approximation of the
first one, leading to a more data-efficient and task-adaptive model. The
optimization tasks of both models are minimax saddle point problems that can be
optimized by adversarial training. To demonstrate the effectiveness of MDANs,
we conduct extensive experiments showing superior adaptation performance on
three real-world datasets: sentiment analysis, digit classification, and
vehicle counting.
Robust Hypothesis Test for Nonlinear Effect with Gaussian Processes
Authors: Jeremiah Zhe Liu, Brent Coull
Date: 2017-10-03 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Gaussian Processes

This work constructs a hypothesis test for detecting whether an
data-generating function $h: R^p \rightarrow R$ belongs to a specific
reproducing kernel Hilbert space $\mathcal{H}_0$ , where the structure of
$\mathcal{H}_0$ is only partially known. Utilizing the theory of reproducing
kernels, we reduce this hypothesis to a simple one-sided score test for a
scalar parameter, develop a testing procedure that is robust against the
mis-specification of kernel functions, and also propose an ensemble-based
estimator for the null model to guarantee test performance in small samples. To
demonstrate the utility of the proposed method, we apply our test to the
problem of detecting nonlinear interaction between groups of continuous
features. We evaluate the finite-sample performance of our test under different
data-generating functions and estimation strategies for the null model. Our
results reveal interesting connections between notions in machine learning
(model underfit/overfit) and those in statistical inference (i.e. Type I
error/power of hypothesis test), and also highlight unexpected consequences of
common model estimating strategies (e.g. estimating kernel hyperparameters
using maximum likelihood estimation) on model inference.
Similarity-based Multi-label Learning
Authors: Ryan A. Rossi, Nesreen K. Ahmed, Hoda Eldardiry, Rong Zhou
Date: 2017-10-27 00:00:00
Tasks: Classification, General Classification, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, Multi-Label Learning

Multi-label classification is an important learning problem with many
applications. In this work, we propose a principled similarity-based approach
for multi-label learning called SML. We also introduce a similarity-based
approach for predicting the label set size. The experimental results
demonstrate the effectiveness of SML for multi-label classification where it is
shown to compare favorably with a wide variety of existing algorithms across a
range of evaluation criterion.
Multi-modal Aggregation for Video Classification
Authors: Chen Chen, Xiaowei Zhao, Yang Liu
Date: 2017-10-27 00:00:00
Tasks: Classification, General Classification, Video Classification

In this paper, we present a solution to Large-Scale Video Classification
Challenge (LSVC2017) [1] that ranked the 1st place. We focused on a variety of
modalities that cover visual, motion and audio. Also, we visualized the
aggregation process to better understand how each modality takes effect. Among
the extracted modalities, we found Temporal-Spatial features calculated by 3D
convolution quite promising that greatly improved the performance. We attained
the official metric mAP 0.8741 on the testing set with the ensemble model.
Lower Bounds for Higher-Order Convex Optimization
Authors: Naman Agarwal, Elad Hazan
Date: 2017-10-27 00:00:00

State-of-the-art methods in convex and non-convex optimization employ
higher-order derivative information, either implicitly or explicitly. We
explore the limitations of higher-order optimization and prove that even for
convex optimization, a polynomial dependence on the approximation guarantee and
higher-order smoothness parameters is necessary. As a special case, we show
Nesterov's accelerated cubic regularization method to be nearly tight.
Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network
Authors: Lixin Fan
Date: 2017-10-27 00:00:00
Proceeding: NeurIPS 2017 12

We revisit fuzzy neural network with a cornerstone notion of generalized
hamming distance, which provides a novel and theoretically justified framework
to re-interpret many useful neural network techniques in terms of fuzzy logic.
In particular, we conjecture and empirically illustrate that, the celebrated
batch normalization (BN) technique actually adapts the normalized bias such
that it approximates the rightful bias induced by the generalized hamming
distance. Once the due bias is enforced analytically, neither the optimization
of bias terms nor the sophisticated batch normalization is needed. Also in the
light of generalized hamming distance, the popular rectified linear units
(ReLU) can be treated as setting a minimal hamming distance threshold between
network inputs and weights. This thresholding scheme, on the one hand, can be
improved by introducing double thresholding on both extremes of neuron outputs.
On the other hand, ReLUs turn out to be non-essential and can be removed from
networks trained for simple tasks like MNIST classification. The proposed
generalized hamming network (GHN) as such not only lends itself to rigorous
analysis and interpretation within the fuzzy logic theory but also demonstrates
fast learning speed, well-controlled behaviour and state-of-the-art
performances on a variety of learning tasks.
Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
Authors: Jiasen Lu, Anitha Kannan, Jianwei Yang, Devi Parikh, Dhruv Batra
Date: 2017-06-05 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Informativeness, Metric Learning, Transfer Learning, Visual Dialog

We present a novel training framework for neural sequence models,
particularly for grounded dialog generation. The standard training paradigm for
these models is maximum likelihood estimation (MLE), or minimizing the
cross-entropy of the human responses. Across a variety of domains, a recurring
problem with MLE trained generative neural dialog models (G) is that they tend
to produce 'safe' and generic responses ("I don't know", "I can't tell"). In
contrast, discriminative dialog models (D) that are trained to rank a list of
candidate human responses outperform their generative counterparts; in terms of
automatic metrics, diversity, and informativeness of the responses. However, D
is not useful in practice since it cannot be deployed to have real
conversations with users.
  Our work aims to achieve the best of both worlds -- the practical usefulness
of G and the strong performance of D -- via knowledge transfer from D to G. Our
primary contribution is an end-to-end trainable generative visual dialog model,
where G receives gradients from D as a perceptual (not adversarial) loss of the
sequence sampled from G. We leverage the recently proposed Gumbel-Softmax (GS)
approximation to the discrete distribution -- specifically, an RNN augmented
with a sequence of GS samplers, coupled with the straight-through gradient
estimator to enable end-to-end differentiability. We also introduce a stronger
encoder for visual dialog, and employ a self-attention mechanism for answer
encoding along with a metric learning loss to aid D in better capturing
semantic similarities in answer responses. Overall, our proposed model
outperforms state-of-the-art on the VisDial dataset by a significant margin
(2.67% on recall@10). The source code can be downloaded from
https://github.com/jiasenlu/visDial.pytorch.
Softmax Q-Distribution Estimation for Structured Prediction: A Theoretical Interpretation for RAML
Authors: Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, Graham Neubig, Eduard Hovy
Date: 2017-05-19 00:00:00
Proceeding: ICLR 2018 1
Tasks: Dependency Parsing, Image Captioning, Machine Translation, Multi-class Classification, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Structured Prediction

Reward augmented maximum likelihood (RAML), a simple and effective learning
framework to directly optimize towards the reward function in structured
prediction tasks, has led to a number of impressive empirical successes. RAML
incorporates task-specific reward by performing maximum-likelihood updates on
candidate outputs sampled according to an exponentiated payoff distribution,
which gives higher probabilities to candidates that are close to the reference
output. While RAML is notable for its simplicity, efficiency, and its
impressive empirical successes, the theoretical properties of RAML, especially
the behavior of the exponentiated payoff distribution, has not been examined
thoroughly. In this work, we introduce softmax Q-distribution estimation, a
novel theoretical interpretation of RAML, which reveals the relation between
RAML and Bayesian decision theory. The softmax Q-distribution can be regarded
as a smooth approximation of the Bayes decision boundary, and the Bayes
decision rule is achieved by decoding with this Q-distribution. We further show
that RAML is equivalent to approximately estimating the softmax Q-distribution,
with the temperature $\tau$ controlling approximation error. We perform two
experiments, one on synthetic data of multi-class classification and one on
real data of image captioning, to demonstrate the relationship between RAML and
the proposed softmax Q-distribution estimation method, verifying our
theoretical analysis. Additional experiments on three structured prediction
tasks with rewards defined on sequential (named entity recognition), tree-based
(dependency parsing) and irregular (machine translation) structures show
notable improvements over maximum likelihood baselines.
A Self-Training Method for Semi-Supervised GANs
Authors: Alan Do-Omri, Dalei Wu, Xiaohua Liu
Date: 2017-10-27 00:00:00
Proceeding: ICLR 2018 1
Tasks: Data Augmentation

Since the creation of Generative Adversarial Networks (GANs), much work has
been done to improve their training stability, their generated image quality,
their range of application but nearly none of them explored their self-training
potential. Self-training has been used before the advent of deep learning in
order to allow training on limited labelled training data and has shown
impressive results in semi-supervised learning. In this work, we combine these
two ideas and make GANs self-trainable for semi-supervised learning tasks by
exploiting their infinite data generation potential. Results show that using
even the simplest form of self-training yields an improvement. We also show
results for a more complex self-training scheme that performs at least as well
as the basic self-training scheme but with significantly less data
augmentation.
OBTAIN: Real-Time Beat Tracking in Audio Signals
Authors: Ali Mottaghi, Kayhan Behdin, Ashkan Esmaeili, Mohammadreza Heydari, Farokh Marvasti
Date: 2017-04-07 00:00:00
Tasks: Beat Tracking, Online Beat Tracking

In this paper, we design a system in order to perform the real-time beat
tracking for an audio signal. We use Onset Strength Signal (OSS) to detect the
onsets and estimate the tempos. Then, we form Cumulative Beat Strength Signal
(CBSS) by taking advantage of OSS and estimated tempos. Next, we perform peak
detection by extracting the periodic sequence of beats among all CBSS peaks. In
simulations, we can see that our proposed algorithm, Online Beat TrAckINg
(OBTAIN), outperforms state-of-art results in terms of prediction accuracy
while maintaining comparable and practical computational complexity. The
real-time performance is tractable visually as illustrated in the simulations.
Principled Hybrids of Generative and Discriminative Domain Adaptation
Authors: Han Zhao, Zhenyao Zhu, Junjie Hu, Adam Coates, Geoff Gordon
Date: 2017-05-25 00:00:00
Proceeding: ICLR 2018 1
Tasks: Domain Adaptation

We propose a probabilistic framework for domain adaptation that blends both
generative and discriminative modeling in a principled way. Under this
framework, generative and discriminative models correspond to specific choices
of the prior over parameters. This provides us a very general way to
interpolate between generative and discriminative extremes through different
choices of priors. By maximizing both the marginal and the conditional
log-likelihoods, models derived from this framework can use both labeled
instances from the source domain as well as unlabeled instances from both
source and target domains. Under this framework, we show that the popular
reconstruction loss of autoencoder corresponds to an upper bound of the
negative marginal log-likelihoods of unlabeled instances, where marginal
distributions are given by proper kernel density estimations. This provides a
way to interpret the empirical success of autoencoders in domain adaptation and
semi-supervised learning. We instantiate our framework using neural networks,
and build a concrete model, DAuto. Empirically, we demonstrate the
effectiveness of DAuto on text, image and speech datasets, showing that it
outperforms related competitors when domain adaptation is possible.
Probability Series Expansion Classifier that is Interpretable by Design
Authors: Sapan Agarwal, Corey M. Hudson
Date: 2017-10-27 00:00:00
Tasks: General Classification

This work presents a new classifier that is specifically designed to be fully
interpretable. This technique determines the probability of a class outcome,
based directly on probability assignments measured from the training data. The
accuracy of the predicted probability can be improved by measuring more
probability estimates from the training data to create a series expansion that
refines the predicted probability. We use this work to classify four standard
datasets and achieve accuracies comparable to that of Random Forests. Because
this technique is interpretable by design, it is capable of determining the
combinations of features that contribute to a particular classification
probability for individual cases as well as the weightings of each of
combination of features.
Searching for Activation Functions
Authors: Prajit Ramachandran, Barret Zoph, Quoc V. Le
Date: 2017-10-16 00:00:00
Proceeding: ICLR 2018 1
Tasks: Image Classification, Reinforcement Learning

The choice of activation functions in deep networks has a significant effect
on the training dynamics and task performance. Currently, the most successful
and widely-used activation function is the Rectified Linear Unit (ReLU).
Although various hand-designed alternatives to ReLU have been proposed, none
have managed to replace it due to inconsistent gains. In this work, we propose
to leverage automatic search techniques to discover new activation functions.
Using a combination of exhaustive and reinforcement learning-based search, we
discover multiple novel activation functions. We verify the effectiveness of
the searches by conducting an empirical evaluation with the best discovered
activation function. Our experiments show that the best discovered activation
function, $f(x) = x \cdot \text{sigmoid}(\beta x)$, which we name Swish, tends
to work better than ReLU on deeper models across a number of challenging
datasets. For example, simply replacing ReLUs with Swish units improves top-1
classification accuracy on ImageNet by 0.9\% for Mobile NASNet-A and 0.6\% for
Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it
easy for practitioners to replace ReLUs with Swish units in any neural network.
Separation of Water and Fat Magnetic Resonance Imaging Signals Using Deep Learning with Convolutional Neural Networks
Authors: James W Goldfarb
Date: 2017-10-27 00:00:00

Purpose: A new method for magnetic resonance (MR) imaging water-fat
separation using a convolutional neural network (ConvNet) and deep learning
(DL) is presented. Feasibility of the method with complex and magnitude images
is demonstrated with a series of patient studies and accuracy of predicted
quantitative values is analyzed.
  Methods: Water-fat separation of 1200 gradient-echo acquisitions from 90
imaging sessions (normal, acute and chronic myocardial infarction) was
performed using a conventional model based method with modeling of R2* and
off-resonance and a multi-peak fat spectrum. A U-Net convolutional neural
network for calculation of water-only, fat-only, R2* and off-resonance images
was trained with 900 gradient-echo Multiple and single-echo complex and
magnitude input data algorithms were studied and compared to conventional
extended echo modeling.
  Results: The U-Net ConvNet was easily trained and provided water-fat
separation results visually comparable to conventional methods. Myocardial fat
deposition in chronic myocardial infarction and intramyocardial hemorrhage in
acute myocardial infarction were well visualized in the DL results. Predicted
values for R2*, off-resonance, water and fat signal intensities were well
correlated with conventional model based water fat separation (R2>=0.97,
p<0.001). DL images had a 14% higher signal-to-noise ratio (p<0.001) when
compared to the conventional method.
  Conclusion: Deep learning utilizing ConvNets is a feasible method for MR
water-fat separationimaging with complex, magnitude and single echo image data.
A trained U-Net can be efficiently used for MR water-fat separation, providing
results comparable to conventional model based methods.
BLENDER: Enabling Local Search with a Hybrid Differential Privacy Model
Authors: Brendan Avent, Aleksandra Korolova, David Zeber, Torgeir Hovden, Benjamin Livshits
Date: 2017-05-02 00:00:00

We propose a hybrid model of differential privacy that considers a combination of regular and opt-in users who desire the differential privacy guarantees of the local privacy model and the trusted curator model, respectively. We demonstrate that within this model, it is possible to design a new type of blended algorithm for the task of privately computing the head of a search log. This blended approach provides significant improvements in the utility of obtained data compared to related work while providing users with their desired privacy guarantees. Specifically, on two large search click data sets, comprising 1.75 and 16 GB respectively, our approach attains NDCG values exceeding 95% across a range of privacy budget values.