pwc-archive/papers-with-abstracts huggingface dataset:
SyncGAN: Synchronize the Latent Space of Cross-modal Generative Adversarial Networks
Authors: Wen-Cheng Chen, Chien-Wen Chen, Min-Chun Hu
Date: 2018-04-02 00:00:00
Tasks: Generative Adversarial Network

Generative adversarial network (GAN) has achieved impressive success on
cross-domain generation, but it faces difficulty in cross-modal generation due
to the lack of a common distribution between heterogeneous data. Most existing
methods of conditional based cross-modal GANs adopt the strategy of
one-directional transfer and have achieved preliminary success on text-to-image
transfer. Instead of learning the transfer between different modalities, we aim
to learn a synchronous latent space representing the cross-modal common
concept. A novel network component named synchronizer is proposed in this work
to judge whether the paired data is synchronous/corresponding or not, which can
constrain the latent space of generators in the GANs. Our GAN model, named as
SyncGAN, can successfully generate synchronous data (e.g., a pair of image and
sound) from identical random noise. For transforming data from one modality to
another, we recover the latent code by inverting the mappings of a generator
and use it to generate data of different modality. In addition, the proposed
model can achieve semi-supervised learning, which makes our model more flexible
for practical applications.
A Note on Kaldi's PLDA Implementation
Authors: Ke Ding
Date: 2018-04-02 00:00:00

Some explanations to Kaldi's PLDA implementation to make formula derivation
easier to catch.
An End-to-end Neural Natural Language Interface for Databases
Authors: Prasetya Utama, Nathaniel Weir, Fuat Basik, Carsten Binnig, Ugur Cetintemel, Benjamin Hättasch, Amir Ilkhechi, Shekar Ramaswamy, Arif Usta
Date: 2018-04-02 00:00:00
Tasks: Decision Making

The ability to extract insights from new data sets is critical for decision
making. Visual interactive tools play an important role in data exploration
since they provide non-technical users with an effective way to visually
compose queries and comprehend the results. Natural language has recently
gained traction as an alternative query interface to databases with the
potential to enable non-expert users to formulate complex questions and
information needs efficiently and effectively. However, understanding natural
language questions and translating them accurately to SQL is a challenging
task, and thus Natural Language Interfaces for Databases (NLIDBs) have not yet
made their way into practical tools and commercial products.
  In this paper, we present DBPal, a novel data exploration tool with a natural
language interface. DBPal leverages recent advances in deep models to make
query understanding more robust in the following ways: First, DBPal uses a deep
model to translate natural language statements to SQL, making the translation
process more robust to paraphrasing and other linguistic variations. Second, to
support the users in phrasing questions without knowing the database schema and
the query features, DBPal provides a learned auto-completion model that
suggests partial query extensions to users during query formulation and thus
helps to write complex queries.
A Lightweight Approach for On-the-Fly Reflectance Estimation
Authors: Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, Jan Kautz
Date: 2017-05-19 00:00:00
Proceeding: ICCV 2017 10
Tasks: Color Constancy

Estimating surface reflectance (BRDF) is one key component for complete 3D
scene capture, with wide applications in virtual reality, augmented reality,
and human computer interaction. Prior work is either limited to controlled
environments (\eg gonioreflectometers, light stages, or multi-camera domes), or
requires the joint optimization of shape, illumination, and reflectance, which
is often computationally too expensive (\eg hours of running time) for
real-time applications. Moreover, most prior work requires HDR images as input
which further complicates the capture process. In this paper, we propose a
lightweight approach for surface reflectance estimation directly from $8$-bit
RGB images in real-time, which can be easily plugged into any 3D
scanning-and-fusion system with a commodity RGBD sensor. Our method is
learning-based, with an inference time of less than 90ms per scene and a model
size of less than 340K bytes. We propose two novel network architectures,
HemiCNN and Grouplet, to deal with the unstructured input data from multiple
viewpoints under unknown illumination. We further design a loss function to
resolve the color-constancy and scale ambiguity. In addition, we have created a
large synthetic dataset, SynBRDF, which comprises a total of $500$K RGBD images
rendered with a physically-based ray tracer under a variety of natural
illumination, covering $5000$ materials and $5000$ shapes. SynBRDF is the first
large-scale benchmark dataset for reflectance estimation. Experiments on both
synthetic data and real data show that the proposed method effectively recovers
surface reflectance, and outperforms prior work for reflectance estimation in
uncontrolled environments.
Generative Spatiotemporal Modeling Of Neutrophil Behavior
Authors: Narita Pandhe, Balazs Rada, Shannon Quinn
Date: 2018-04-02 00:00:00

Cell motion and appearance have a strong correlation with cell cycle and
disease progression. Many contemporary efforts in machine learning utilize
spatio-temporal models to predict a cell's physical state and, consequently,
the advancement of disease. Alternatively, generative models learn the
underlying distribution of the data, creating holistic representations that can
be used in learning. In this work, we propose an aggregate model that combine
Generative Adversarial Networks (GANs) and Autoregressive (AR) models to
predict cell motion and appearance in human neutrophils imaged by differential
interference contrast (DIC) microscopy. We bifurcate the task of learning cell
statistics by leveraging GANs for the spatial component and AR models for the
temporal component. The aggregate model learned results offer a promising
computational environment for studying changes in organellar shape, quantity,
and spatial distribution over large sequences.
Low-Latency Video Semantic Segmentation
Authors: Yule Li, Jianping Shi, Dahua Lin
Date: 2018-04-02 00:00:00
Proceeding: CVPR 2018 6
Tasks: Autonomous Driving, Segmentation, Semantic Segmentation, Video Segmentation, Video Semantic Segmentation

Recent years have seen remarkable progress in semantic segmentation. Yet, it
remains a challenging task to apply segmentation techniques to video-based
applications. Specifically, the high throughput of video streams, the sheer
cost of running fully convolutional networks, together with the low-latency
requirements in many real-world applications, e.g. autonomous driving, present
a significant challenge to the design of the video segmentation framework. To
tackle this combined challenge, we develop a framework for video semantic
segmentation, which incorporates two novel components: (1) a feature
propagation module that adaptively fuses features over time via spatially
variant convolution, thus reducing the cost of per-frame computation; and (2)
an adaptive scheduler that dynamically allocate computation based on accuracy
prediction. Both components work together to ensure low latency while
maintaining high segmentation quality. On both Cityscapes and CamVid, the
proposed framework obtained competitive performance compared to the state of
the art, while substantially reducing the latency, from 360 ms to 119 ms.
A Novel Learnable Dictionary Encoding Layer for End-to-End Language Identification
Authors: Weicheng Cai, Zexin Cai, Xiang Zhang, Xiaoqi Wang, Ming Li
Date: 2018-04-02 00:00:00
Tasks: Language Identification

A novel learnable dictionary encoding layer is proposed in this paper for
end-to-end language identification. It is inline with the conventional GMM
i-vector approach both theoretically and practically. We imitate the mechanism
of traditional GMM training and Supervector encoding procedure on the top of
CNN. The proposed layer can accumulate high-order statistics from
variable-length input sequence and generate an utterance level
fixed-dimensional vector representation. Unlike the conventional methods, our
new approach provides an end-to-end learning framework, where the inherent
dictionary are learned directly from the loss function. The dictionaries and
the encoding representation for the classifier are learned jointly. The
representation is orderless and therefore appropriate for language
identification. We conducted a preliminary experiment on NIST LRE07 closed-set
task, and the results reveal that our proposed dictionary encoding layer
achieves significant error reduction comparing with the simple average pooling.
On Kernel Method-Based Connectionist Models and Supervised Deep Learning Without Backpropagation
Authors: Shiyu Duan, Shujian Yu, Yun-Mei Chen, Jose Principe
Date: 2018-02-11 00:00:00
Proceeding: ICLR 2019 5

We propose a novel family of connectionist models based on kernel machines and consider the problem of learning layer-by-layer a compositional hypothesis class, i.e., a feedforward, multilayer architecture, in a supervised setting. In terms of the models, we present a principled method to "kernelize" (partly or completely) any neural network (NN). With this method, we obtain a counterpart of any given NN that is powered by kernel machines instead of neurons. In terms of learning, when learning a feedforward deep architecture in a supervised setting, one needs to train all the components simultaneously using backpropagation (BP) since there are no explicit targets for the hidden layers (Rumelhart86). We consider without loss of generality the two-layer case and present a general framework that explicitly characterizes a target for the hidden layer that is optimal for minimizing the objective function of the network. This characterization then makes possible a purely greedy training scheme that learns one layer at a time, starting from the input layer. We provide realizations of the abstract framework under certain architectures and objective functions. Based on these realizations, we present a layer-wise training algorithm for an l-layer feedforward network for classification, where l>=2 can be arbitrary. This algorithm can be given an intuitive geometric interpretation that makes the learning dynamics transparent. Empirical results are provided to complement our theory. We show that the kernelized networks, trained layer-wise, compare favorably with classical kernel machines as well as other connectionist models trained by BP. We also visualize the inner workings of the greedy kernelized models to validate our claim on the transparency of the layer-wise algorithm.
Attention-based Ensemble for Deep Metric Learning
Authors: Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, Keunjoo Kwon
Date: 2018-04-02 00:00:00
Proceeding: ECCV 2018 9
Tasks: Diversity, Image Retrieval, Metric Learning, Retrieval

Deep metric learning aims to learn an embedding function, modeled as deep
neural network. This embedding function usually puts semantically similar
images close while dissimilar images far from each other in the learned
embedding space. Recently, ensemble has been applied to deep metric learning to
yield state-of-the-art results. As one important aspect of ensemble, the
learners should be diverse in their feature embeddings. To this end, we propose
an attention-based ensemble, which uses multiple attention masks, so that each
learner can attend to different parts of the object. We also propose a
divergence loss, which encourages diversity among the learners. The proposed
method is applied to the standard benchmarks of deep metric learning and
experimental results show that it outperforms the state-of-the-art methods by a
significant margin on image retrieval tasks.
Insights into End-to-End Learning Scheme for Language Identification
Authors: Weicheng Cai, Zexin Cai, Wenbo Liu, Xiaoqi Wang, Ming Li
Date: 2018-04-02 00:00:00
Tasks: Language Identification

A novel interpretable end-to-end learning scheme for language identification
is proposed. It is in line with the classical GMM i-vector methods both
theoretically and practically. In the end-to-end pipeline, a general encoding
layer is employed on top of the front-end CNN, so that it can encode the
variable-length input sequence into an utterance level vector automatically.
After comparing with the state-of-the-art GMM i-vector methods, we give
insights into CNN, and reveal its role and effect in the whole pipeline. We
further introduce a general encoding layer, illustrating the reason why they
might be appropriate for language identification. We elaborate on several
typical encoding layers, including a temporal average pooling layer, a
recurrent encoding layer and a novel learnable dictionary encoding layer. We
conducted experiment on NIST LRE07 closed-set task, and the results show that
our proposed end-to-end systems achieve state-of-the-art performance.
Efficient and Deep Person Re-Identification using Multi-Level Similarity
Authors: Yiluan Guo, Ngai-Man Cheung
Date: 2018-03-30 00:00:00
Proceeding: CVPR 2018 6
Tasks: Person Re-Identification

Person Re-Identification (ReID) requires comparing two images of person
captured under different conditions. Existing work based on neural networks
often computes the similarity of feature maps from one single convolutional
layer. In this work, we propose an efficient, end-to-end fully convolutional
Siamese network that computes the similarities at multiple levels. We
demonstrate that multi-level similarity can improve the accuracy considerably
using low-complexity network structures in ReID problem. Specifically, first,
we use several convolutional layers to extract the features of two input
images. Then, we propose Convolution Similarity Network to compute the
similarity score maps for the inputs. We use spatial transformer networks
(STNs) to determine spatial attention. We propose to apply efficient depth-wise
convolution to compute the similarity. The proposed Convolution Similarity
Networks can be inserted into different convolutional layers to extract visual
similarities at different levels. Furthermore, we use an improved ranking loss
to further improve the performance. Our work is the first to propose to compute
visual similarities at low, middle and high levels for ReID. With extensive
experiments and analysis, we demonstrate that our system, compact yet
effective, can achieve competitive results with much smaller model size and
computational complexity.
Recall Traces: Backtracking Models for Efficient Reinforcement Learning
Authors: Anirudh Goyal, Philemon Brakel, William Fedus, Soumye Singhal, Timothy Lillicrap, Sergey Levine, Hugo Larochelle, Yoshua Bengio
Date: 2018-04-02 00:00:00
Proceeding: ICLR 2019 5
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In many environments only a tiny subset of all states yield high reward. In
these cases, few of the interactions with the environment provide a relevant
learning signal. Hence, we may want to preferentially train on those
high-reward states and the probable trajectories leading to them. To this end,
we advocate for the use of a backtracking model that predicts the preceding
states that terminate at a given high-reward state. We can train a model which,
starting from a high value state (or one that is estimated to have high value),
predicts and sample for which the (state, action)-tuples may have led to that
high value state. These traces of (state, action) pairs, which we refer to as
Recall Traces, sampled from this backtracking model starting from a high value
state, are informative as they terminate in good states, and hence we can use
these traces to improve a policy. We provide a variational interpretation for
this idea and a practical algorithm in which the backtracking model samples
from an approximate posterior distribution over trajectories which lead to
large rewards. Our method improves the sample efficiency of both on- and
off-policy RL algorithms across several environments and tasks.
End-to-End Detection and Re-identification Integrated Net for Person Search
Authors: Zhenwei He, Lei Zhang, Wei Jia
Date: 2018-04-02 00:00:00
Tasks: Pedestrian Detection, Person Search

This paper proposes a pedestrian detection and re-identification (re-id)
integration net (I-Net) in an end-to-end learning framework. The I-Net is used
in real-world video surveillance scenarios, where the target person needs to be
searched in the whole scene videos, while the annotations of pedestrian
bounding boxes are unavailable. By comparing to the OIM which is a work for
joint detection and re-id, we have three distinct contributions. First, we
introduce a Siamese architecture of I-Net instead of 1 stream, such that a
verification task can be implemented. Second, we propose a novel on-line
pairing loss (OLP) and hard example priority softmax loss (HEP), such that only
the hard negatives are posed much attention in loss computation. Third, an
on-line dictionary for negative samples storage is designed in I-Net without
recording the positive samples. We show our result on person search datasets,
the gap between detection and re-identification is narrowed. The superior
performance can be achieved.
Geometry-Aware Learning of Maps for Camera Localization
Authors: Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, Jan Kautz
Date: 2017-12-09 00:00:00
Proceeding: CVPR 2018 6
Tasks: Camera Localization, Visual Localization

Maps are a key component in image-based camera localization and visual SLAM
systems: they are used to establish geometric constraints between images,
correct drift in relative pose estimation, and relocalize cameras after lost
tracking. The exact definitions of maps, however, are often
application-specific and hand-crafted for different scenarios (e.g. 3D
landmarks, lines, planes, bags of visual words). We propose to represent maps
as a deep neural net called MapNet, which enables learning a data-driven map
representation. Unlike prior work on learning maps, MapNet exploits cheap and
ubiquitous sensory inputs like visual odometry and GPS in addition to images
and fuses them together for camera localization. Geometric constraints
expressed by these inputs, which have traditionally been used in bundle
adjustment or pose-graph optimization, are formulated as loss terms in MapNet
training and also used during inference. In addition to directly improving
localization accuracy, this allows us to update the MapNet (i.e., maps) in a
self-supervised manner using additional unlabeled video sequences from the
scene. We also propose a novel parameterization for camera rotation which is
better suited for deep-learning based camera pose regression. Experimental
results on both the indoor 7-Scenes dataset and the outdoor Oxford RobotCar
dataset show significant performance improvement over prior work. The MapNet
project webpage is https://goo.gl/mRB3Au.
TipsC: Tips and Corrections for programming MOOCs
Authors: Saksham Sharma, Pallav Agarwal, Parv Mor, Amey Karkare
Date: 2018-04-02 00:00:00
Tasks: Clustering

With the widespread adoption of MOOCs in academic institutions, it has become
imperative to come up with better techniques to solve the tutoring and grading
problems posed by programming courses. Programming being the new 'writing', it
becomes a challenge to ensure that a large section of the society is exposed to
programming. Due to the gradient in learning abilities of students, the course
instructor must ensure that everyone can cope up with the material, and receive
adequate help in completing assignments while learning along the way. We
introduce TipsC for this task. By analyzing a large number of correct
submissions, TipsC can search for correct codes resembling a given incorrect
solution. Without revealing the actual code, TipsC then suggests changes in the
incorrect code to help the student fix logical runtime errors. In addition,
this also serves as a cluster visualization tool for the instructor, revealing
different patterns in user submissions. We evaluated the effectiveness of
TipsC's clustering algorithm on data collected from previous offerings of an
introductory programming course conducted at IIT Kanpur where the grades were
given by human TAs. The results show the weighted average variance of marks for
clusters when similar submissions are grouped together is 47% less compared to
the case when all programs are grouped together.
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
Authors: Kensho Hara, Hirokatsu Kataoka, Yutaka Satoh
Date: 2017-11-27 00:00:00
Proceeding: CVPR 2018 6
Tasks: Action Recognition

The purpose of this study is to determine whether current video datasets have
sufficient data for training very deep convolutional neural networks (CNNs)
with spatio-temporal three-dimensional (3D) kernels. Recently, the performance
levels of 3D CNNs in the field of action recognition have improved
significantly. However, to date, conventional research has only explored
relatively shallow 3D architectures. We examine the architectures of various 3D
CNNs from relatively shallow to very deep ones on current video datasets. Based
on the results of those experiments, the following conclusions could be
obtained: (i) ResNet-18 training resulted in significant overfitting for
UCF-101, HMDB-51, and ActivityNet but not for Kinetics. (ii) The Kinetics
dataset has sufficient data for training of deep 3D CNNs, and enables training
of up to 152 ResNets layers, interestingly similar to 2D ResNets on ImageNet.
ResNeXt-101 achieved 78.4% average accuracy on the Kinetics test set. (iii)
Kinetics pretrained simple 3D architectures outperforms complex 2D
architectures, and the pretrained ResNeXt-101 achieved 94.5% and 70.2% on
UCF-101 and HMDB-51, respectively. The use of 2D CNNs trained on ImageNet has
produced significant progress in various tasks in image. We believe that using
deep 3D CNNs together with Kinetics will retrace the successful history of 2D
CNNs and ImageNet, and stimulate advances in computer vision for videos. The
codes and pretrained models used in this study are publicly available.
https://github.com/kenshohara/3D-ResNets-PyTorch
Learning Structural Weight Uncertainty for Sequential Decision-Making
Authors: Ruiyi Zhang, Chunyuan Li, Changyou Chen, Lawrence Carin
Date: 2017-12-30 00:00:00
Tasks: Decision Making, Multi-Armed Bandits, Reinforcement Learning, Reinforcement Learning (RL), Sequential Decision Making

Learning probability distributions on the weights of neural networks (NNs)
has recently proven beneficial in many applications. Bayesian methods, such as
Stein variational gradient descent (SVGD), offer an elegant framework to reason
about NN model uncertainty. However, by assuming independent Gaussian priors
for the individual NN weights (as often applied), SVGD does not impose prior
knowledge that there is often structural information (dependence) among
weights. We propose efficient posterior learning of structural weight
uncertainty, within an SVGD framework, by employing matrix variate Gaussian
priors on NN parameters. We further investigate the learned structural
uncertainty in sequential decision-making problems, including contextual
bandits and reinforcement learning. Experiments on several synthetic and real
datasets indicate the superiority of our model, compared with state-of-the-art
methods.