pwc-archive/papers-with-abstracts huggingface dataset:
Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks
Authors: Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, Shih-Fu Chang
Date: 2017-12-05 00:00:00
Proceeding: CVPR 2018 6
Tasks: General Classification, Zero-Shot Learning

We propose a novel framework called Semantics-Preserving Adversarial
Embedding Network (SP-AEN) for zero-shot visual recognition (ZSL), where test
images and their classes are both unseen during training. SP-AEN aims to tackle
the inherent problem --- semantic loss --- in the prevailing family of
embedding-based ZSL, where some semantics would be discarded during training if
they are non-discriminative for training classes, but could become critical for
recognizing test classes. Specifically, SP-AEN prevents the semantic loss by
introducing an independent visual-to-semantic space embedder which disentangles
the semantic space into two subspaces for the two arguably conflicting
objectives: classification and reconstruction. Through adversarial learning of
the two subspaces, SP-AEN can transfer the semantics from the reconstructive
subspace to the discriminative one, accomplishing the improved zero-shot
recognition of unseen classes. Comparing with prior works, SP-AEN can not only
improve classification but also generate photo-realistic images, demonstrating
the effectiveness of semantic preservation. On four popular benchmarks: CUB,
AWA, SUN and aPY, SP-AEN considerably outperforms other state-of-the-art
methods by an absolute performance difference of 12.2\%, 9.3\%, 4.0\%, and
3.6\% in terms of harmonic mean values
Snap Angle Prediction for 360$^{\circ}$ Panoramas
Authors: Bo Xiong, Kristen Grauman
Date: 2018-03-31 00:00:00
Tasks: Prediction, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

360$^{\circ}$ panoramas are a rich medium, yet notoriously difficult to
visualize in the 2D image plane. We explore how intelligent rotations of a
spherical image may enable content-aware projection with fewer perceptible
distortions. Whereas existing approaches assume the viewpoint is fixed,
intuitively some viewing angles within the sphere preserve high-level objects
better than others. To discover the relationship between these optimal snap
angles and the spherical panorama's content, we develop a reinforcement
learning approach for the cubemap projection model. Implemented as a deep
recurrent neural network, our method selects a sequence of rotation actions and
receives reward for avoiding cube boundaries that overlap with important
foreground objects. We show our approach creates more visually pleasing
panoramas while using 5x less computation than the baseline.
Grounding Referring Expressions in Images by Variational Context
Authors: Hanwang Zhang, Yulei Niu, Shih-Fu Chang
Date: 2017-12-05 00:00:00
Proceeding: CVPR 2018 6
Tasks: Multiple Instance Learning, Referring Expression

We focus on grounding (i.e., localizing or linking) referring expressions in
images, e.g., "largest elephant standing behind baby elephant". This is a
general yet challenging vision-language task since it does not only require the
localization of objects, but also the multimodal comprehension of context ---
visual attributes (e.g., "largest", "baby") and relationships (e.g., "behind")
that help to distinguish the referent from other objects, especially those of
the same category. Due to the exponential complexity involved in modeling the
context associated with multiple image regions, existing work oversimplifies
this task to pairwise region modeling by multiple instance learning. In this
paper, we propose a variational Bayesian method, called Variational Context, to
solve the problem of complex context modeling in referring expression
grounding. Our model exploits the reciprocal relation between the referent and
context, i.e., either of them influences the estimation of the posterior
distribution of the other, and thereby the search space of context can be
greatly reduced, resulting in better localization of referent. We develop a
novel cue-specific language-vision embedding network that learns this
reciprocity model end-to-end. We also extend the model to the unsupervised
setting where no annotation for the referent is available. Extensive
experiments on various benchmarks show consistent improvement over
state-of-the-art methods in both supervised and unsupervised settings.
Quantitative Evaluation of Style Transfer
Authors: Mao-Chuang Yeh, Shuai Tang, Anand Bhattad, D. A. Forsyth
Date: 2018-03-31 00:00:00
Tasks: Style Transfer

Style transfer methods produce a transferred image which is a rendering of a
content image in the manner of a style image. There is a rich literature of
variant methods. However, evaluation procedures are qualitative, mostly
involving user studies. We describe a novel quantitative evaluation procedure.
One plots effectiveness (a measure of the extent to which the style was
transferred) against coherence (a measure of the extent to which the
transferred image decomposes into objects in the same way that the content
image does) to obtain an EC plot.
  We construct EC plots comparing a number of recent style transfer methods.
Most methods control within-layer gram matrices, but we also investigate a
method that controls cross-layer gram matrices. These EC plots reveal a number
of intriguing properties of recent style transfer methods. The style used has a
strong effect on the outcome, for all methods. Using large style weights does
not necessarily improve effectiveness, and can produce worse results.
Cross-layer gram matrices easily beat all other methods, but some styles remain
difficult for all methods. Ensemble methods show real promise. It is likely
that, for current methods, each style requires a different choice of weights to
obtain the best results, so that automated weight setting methods are
desirable. Finally, we show evidence comparing our EC evaluations to human
evaluations.
Multi-label Learning with Missing Labels using Mixed Dependency Graphs
Authors: Baoyuan Wu, Fan Jia, Wei Liu, Bernard Ghanem, Siwei Lyu
Date: 2018-03-31 00:00:00
Tasks: Image Retrieval, Missing Labels, Multi-Label Learning, Retrieval, TAG

This work focuses on the problem of multi-label learning with missing labels
(MLML), which aims to label each test instance with multiple class labels given
training instances that have an incomplete/partial set of these labels. The key
point to handle missing labels is propagating the label information from
provided labels to missing labels, through a dependency graph that each label
of each instance is treated as a node. We build this graph by utilizing
different types of label dependencies. Specifically, the instance-level
similarity is served as undirected edges to connect the label nodes across
different instances and the semantic label hierarchy is used as directed edges
to connect different classes. This base graph is referred to as the mixed
dependency graph, as it includes both undirected and directed edges.
Furthermore, we present another two types of label dependencies to connect the
label nodes across different classes. One is the class co-occurrence, which is
also encoded as undirected edges. Combining with the base graph, we obtain a
new mixed graph, called MG-CO (mixed graph with co-occurrence). The other is
the sparse and low rank decomposition of the whole label matrix, to embed
high-order dependencies over all labels. Combining with the base graph, the new
mixed graph is called as MG-SL (mixed graph with sparse and low rank
decomposition). Based on MG-CO and MG-SL, we propose two convex transductive
formulations of the MLML problem, denoted as MLMG-CO and MLMG-SL, respectively.
Two important applications, including image annotation and tag based image
retrieval, can be jointly handled using our proposed methods. Experiments on
benchmark datasets show that our methods give significant improvements in
performance and robustness to missing labels over the state-of-the-art methods.
Tagging like Humans: Diverse and Distinct Image Annotation
Authors: Baoyuan Wu, Weidong Chen, Peng Sun, Wei Liu, Bernard Ghanem, Siwei Lyu
Date: 2018-03-31 00:00:00
Proceeding: CVPR 2018 6
Tasks: Generative Adversarial Network, TAG

In this work we propose a new automatic image annotation model, dubbed {\bf
diverse and distinct image annotation} (D2IA). The generative model D2IA is
inspired by the ensemble of human annotations, which create semantically
relevant, yet distinct and diverse tags. In D2IA, we generate a relevant and
distinct tag subset, in which the tags are relevant to the image contents and
semantically distinct to each other, using sequential sampling from a
determinantal point process (DPP) model. Multiple such tag subsets that cover
diverse semantic aspects or diverse semantic levels of the image contents are
generated by randomly perturbing the DPP sampling process. We leverage a
generative adversarial network (GAN) model to train D2IA. Extensive experiments
including quantitative and qualitative comparisons, as well as human subject
studies, on two benchmark datasets demonstrate that the proposed model can
produce more diverse and distinct tags than the state-of-the-arts.
Detection under Privileged Information
Authors: Z. Berkay Celik, Patrick McDaniel, Rauf Izmailov, Nicolas Papernot, Ryan Sheatsley, Raquel Alvarez, Ananthram Swami
Date: 2016-03-31 00:00:00
Tasks: Face Recognition, Malware Classification, Transfer Learning

For well over a quarter century, detection systems have been driven by models
learned from input features collected from real or simulated environments. An
artifact (e.g., network event, potential malware sample, suspicious email) is
deemed malicious or non-malicious based on its similarity to the learned model
at runtime. However, the training of the models has been historically limited
to only those features available at runtime. In this paper, we consider an
alternate learning approach that trains models using "privileged"
information--features available at training time but not at runtime--to improve
the accuracy and resilience of detection systems. In particular, we adapt and
extend recent advances in knowledge transfer, model influence, and distillation
to enable the use of forensic or other data unavailable at runtime in a range
of security domains. An empirical evaluation shows that privileged information
increases precision and recall over a system with no privileged information: we
observe up to 7.7% relative decrease in detection error for fast-flux bot
detection, 8.6% for malware traffic detection, 7.3% for malware classification,
and 16.9% for face recognition. We explore the limitations and applications of
different privileged information techniques in detection systems. Such
techniques provide a new means for detection systems to learn from data that
would otherwise not be available at runtime.
Visual Question Reasoning on General Dependency Tree
Authors: Qingxing Cao, Xiaodan Liang, Bailing Li, Guanbin Li, Liang Lin
Date: 2018-03-31 00:00:00
Proceeding: CVPR 2018 6
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA)

The collaborative reasoning for understanding each image-question pair is
very critical but under-explored for an interpretable Visual Question Answering
(VQA) system. Although very recent works also tried the explicit compositional
processes to assemble multiple sub-tasks embedded in the questions, their
models heavily rely on the annotations or hand-crafted rules to obtain valid
reasoning layout, leading to either heavy labor or poor performance on
composition reasoning. In this paper, to enable global context reasoning for
better aligning image and language domains in diverse and unrestricted cases,
we propose a novel reasoning network called Adversarial Composition Modular
Network (ACMN). This network comprises of two collaborative modules: i) an
adversarial attention module to exploit the local visual evidence for each word
parsed from the question; ii) a residual composition module to compose the
previously mined evidence. Given a dependency parse tree for each question, the
adversarial attention module progressively discovers salient regions of one
word by densely combining regions of child word nodes in an adversarial manner.
Then residual composition module merges the hidden representations of an
arbitrary number of children through sum pooling and residual connection. Our
ACMN is thus capable of building an interpretable VQA system that gradually
dives the image cues following a question-driven reasoning route and makes
global reasoning by incorporating the learned knowledge of all attention
modules in a principled manner. Experiments on relational datasets demonstrate
the superiority of our ACMN and visualization results show the explainable
capability of our reasoning system.
A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving
Authors: Xiangyu Yue, Bichen Wu, Sanjit A. Seshia, Kurt Keutzer, Alberto L. Sangiovanni-Vincentelli
Date: 2018-03-31 00:00:00
Tasks: Autonomous Driving, Point Cloud Segmentation, Sensor Fusion

3D LiDAR scanners are playing an increasingly important role in autonomous
driving as they can generate depth information of the environment. However,
creating large 3D LiDAR point cloud datasets with point-level labels requires a
significant amount of manual annotation. This jeopardizes the efficient
development of supervised deep learning algorithms which are often data-hungry.
We present a framework to rapidly create point clouds with accurate point-level
labels from a computer game. The framework supports data collection from both
auto-driving scenes and user-configured scenes. Point clouds from auto-driving
scenes can be used as training data for deep learning algorithms, while point
clouds from user-configured scenes can be used to systematically test the
vulnerability of a neural network, and use the falsifying examples to make the
neural network more robust through retraining. In addition, the scene images
can be captured simultaneously in order for sensor fusion tasks, with a method
proposed to do automatic calibration between the point clouds and captured
scene images. We show a significant improvement in accuracy (+9%) in point
cloud segmentation by augmenting the training dataset with the generated
synthesized data. Our experiments also show by testing and retraining the
network using point clouds from user-configured scenes, the weakness/blind
spots of the neural network can be fixed.
Graph Convolutional Neural Networks via Scattering
Authors: Dongmian Zou, Gilad Lerman
Date: 2018-03-31 00:00:00
Tasks: Node Classification

We generalize the scattering transform to graphs and consequently construct a
convolutional neural network on graphs. We show that under certain conditions,
any feature generated by such a network is approximately invariant to
permutations and stable to graph manipulations. Numerical results demonstrate
competitive performance on relevant datasets.
Adversarial Attacks and Defences Competition
Authors: Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, Jian-Yu Wang, Zhishuai Zhang, Zhou Ren, Alan Yuille, Sangxia Huang, Yao Zhao, Yuzhe Zhao, Zhonglin Han, Junjiajia Long, Yerkebulan Berdibekov, Takuya Akiba, Seiya Tokui, Motoki Abe
Date: 2018-03-31 00:00:00
Tasks: BIG-bench Machine Learning

To accelerate research on adversarial examples and robustness of machine
learning classifiers, Google Brain organized a NIPS 2017 competition that
encouraged researchers to develop new methods to generate adversarial examples
as well as to develop new ways to defend against them. In this chapter, we
describe the structure and organization of the competition and the solutions
developed by several of the top-placing teams.
Iterative Learning with Open-set Noisy Labels
Authors: Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, Shu-Tao Xia
Date: 2018-03-31 00:00:00
Proceeding: CVPR 2018 6

Large-scale datasets possessing clean label annotations are crucial for
training Convolutional Neural Networks (CNNs). However, labeling large-scale
data can be very costly and error-prone, and even high-quality datasets are
likely to contain noisy (incorrect) labels. Existing works usually employ a
closed-set assumption, whereby the samples associated with noisy labels possess
a true class contained within the set of known classes in the training data.
However, such an assumption is too restrictive for many applications, since
samples associated with noisy labels might in fact possess a true class that is
not present in the training data. We refer to this more complex scenario as the
\textbf{open-set noisy label} problem and show that it is nontrivial in order
to make accurate predictions. To address this problem, we propose a novel
iterative learning framework for training CNNs on datasets with open-set noisy
labels. Our approach detects noisy labels and learns deep discriminative
features in an iterative fashion. To benefit from the noisy label detection, we
design a Siamese network to encourage clean labels and noisy labels to be
dissimilar. A reweighting module is also applied to simultaneously emphasize
the learning from clean labels and reduce the effect caused by noisy labels.
Experiments on CIFAR-10, ImageNet and real-world noisy (web-search) datasets
demonstrate that our proposed model can robustly train CNNs in the presence of
a high proportion of open-set as well as closed-set noisy labels.
FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans
Authors: Chen Liu, Jiaye Wu, Yasutaka Furukawa
Date: 2018-03-31 00:00:00
Proceeding: ECCV 2018 9
Tasks: Spatial Reasoning, Vector Graphics

The ultimate goal of this indoor mapping research is to automatically
reconstruct a floorplan simply by walking through a house with a smartphone in
a pocket. This paper tackles this problem by proposing FloorNet, a novel deep
neural architecture. The challenge lies in the processing of RGBD streams
spanning a large 3D space. FloorNet effectively processes the data through
three neural network branches: 1) PointNet with 3D points, exploiting the 3D
information; 2) CNN with a 2D point density image in a top-down view, enhancing
the local spatial reasoning; and 3) CNN with RGB images, utilizing the full
image information. FloorNet exchanges intermediate features across the branches
to exploit the best of all the architectures. We have created a benchmark for
floorplan reconstruction by acquiring RGBD video streams for 155 residential
houses or apartments with Google Tango phones and annotating complete floorplan
information. Our qualitative and quantitative evaluations demonstrate that the
fusion of three branches effectively improves the reconstruction quality. We
hope that the paper together with the benchmark will be an important step
towards solving a challenging vector-graphics reconstruction problem. Code and
data are available at https://github.com/art-programmer/FloorNet.
Characterizing Interconnections and Linguistic Patterns in Twitter
Authors: Johnnatan Messias
Date: 2018-03-30 00:00:00

Social media is considered a democratic space in which people connect and
interact with each other regardless of their gender, race, or any other
demographic aspect. Despite numerous efforts that explore demographic aspects
in social media, it is still unclear whether social media perpetuates old
inequalities from the offline world. In this dissertation, we attempt to
identify gender and race of Twitter users located in the United States using
advanced image processing algorithms from Face++. We investigate how different
demographic groups connect with each other and differentiate them regarding
linguistic styles and also their interests. We quantify to what extent one
group follows and interacts with each other and the extent to which these
connections and interactions reflect in inequalities in Twitter. We also
extract linguistic features from six categories (affective attributes,
cognitive attributes, lexical density and awareness, temporal references,
social and personal concerns, and interpersonal focus) in order to identify the
similarities and the differences in the messages they share in Twitter.
Furthermore, we extract the absolute ranking difference of top phrases between
demographic groups. As a dimension of diversity, we use the topics of interest
that we retrieve from each user. Our analysis shows that users identified as
white and male tend to attain higher positions, in terms of the number of
followers and number of times in another user's lists, in Twitter. There are
clear differences in the way of writing across different demographic groups in
both gender and race domains as well as in the topic of interest. We hope our
effort can stimulate the development of new theories of demographic information
in the online space. Finally, we developed a Web-based system that leverages
the demographic aspects of users to provide transparency to the Twitter
trending topics system.
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
Authors: Sandeep Subramanian, Adam Trischler, Yoshua Bengio, Christopher J. Pal
Date: 2018-03-30 00:00:00
Proceeding: ICLR 2018 1
Tasks: Multi-Task Learning, Natural Language Inference, Paraphrase Identification, Semantic Textual Similarity, Sentence, Transfer Learning

A lot of the recent success in natural language processing (NLP) has been
driven by distributed vector representations of words trained on large amounts
of text in an unsupervised manner. These representations are typically used as
general purpose features for words across a range of NLP problems. However,
extending this success to learning representations of sequences of words, such
as sentences, remains an open problem. Recent work has explored unsupervised as
well as supervised learning techniques with different training objectives to
learn general purpose fixed-length sentence representations. In this work, we
present a simple, effective multi-task learning framework for sentence
representations that combines the inductive biases of diverse training
objectives in a single model. We train this model on several data sources with
multiple training objectives on over 100 million sentences. Extensive
experiments demonstrate that sharing a single recurrent sentence encoder across
weakly related tasks leads to consistent improvements over previous methods. We
present substantial improvements in the context of transfer learning and
low-resource settings using our learned general-purpose representations.
Learning to Organize Knowledge and Answer Questions with N-Gram Machines
Authors: Fan Yang, Jiazhong Nie, William W. Cohen, Ni Lao
Date: 2017-11-17 00:00:00
Tasks: Open-Domain Question Answering, Question Answering

Though deep neural networks have great success in natural language
processing, they are limited at more knowledge intensive AI tasks, such as
open-domain Question Answering (QA). Existing end-to-end deep QA models need to
process the entire text after observing the question, and therefore their
complexity in responding a question is linear in the text size. This is
prohibitive for practical tasks such as QA from Wikipedia, a novel, or the Web.
We propose to solve this scalability issue by using symbolic meaning
representations, which can be indexed and retrieved efficiently with complexity
that is independent of the text size. We apply our approach, called the N-Gram
Machine (NGM), to three representative tasks. First as proof-of-concept, we
demonstrate that NGM successfully solves the bAbI tasks of synthetic text.
Second, we show that NGM scales to large corpus by experimenting on "life-long
bAbI", a special version of bAbI that contains millions of sentences. Lastly on
the WikiMovies dataset, we use NGM to induce latent structure (i.e. schema) and
answer questions from natural language Wikipedia text, with only QA pairs as
weak supervision.