pwc-archive/papers-with-abstracts huggingface dataset:
Learning Residual Images for Face Attribute Manipulation
Authors: Wei Shen, Rujie Liu
Date: 2016-12-16 00:00:00
Proceeding: CVPR 2017 7
Tasks: Attribute, Generative Adversarial Network

Face attributes are interesting due to their detailed description of human
faces. Unlike prior researches working on attribute prediction, we address an
inverse and more challenging problem called face attribute manipulation which
aims at modifying a face image according to a given attribute value. Instead of
manipulating the whole image, we propose to learn the corresponding residual
image defined as the difference between images before and after the
manipulation. In this way, the manipulation can be operated efficiently with
modest pixel modification. The framework of our approach is based on the
Generative Adversarial Network. It consists of two image transformation
networks and a discriminative network. The transformation networks are
responsible for the attribute manipulation and its dual operation and the
discriminative network is used to distinguish the generated images from real
images. We also apply dual learning to allow transformation networks to learn
from each other. Experiments show that residual images can be effectively
learned and used for attribute manipulations. The generated images remain most
of the details in attribute-irrelevant areas.
Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
Authors: Vahid Kazemi, Ali Elqursh
Date: 2017-04-11 00:00:00
Tasks: Visual Question Answering, Visual Question Answering (VQA)

This paper presents a new baseline for visual question answering task. Given
an image and a question in natural language, our model produces accurate
answers according to the content of the image. Our model, while being
architecturally simple and relatively small in terms of trainable parameters,
sets a new state of the art on both unbalanced and balanced VQA benchmark. On
VQA 1.0 open ended challenge, our model achieves 64.6% accuracy on the
test-standard set without using additional data, an improvement of 0.4% over
state of the art, and on newly released VQA 2.0, our model scores 59.7% on
validation set outperforming best previously reported results by 0.5%. The
results presented in this paper are especially interesting because very similar
models have been tried before but significantly lower performance were
reported. In light of the new results we hope to see more meaningful research
on visual question answering in the future.
SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
Authors: Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, Tat-Seng Chua
Date: 2016-11-17 00:00:00
Proceeding: CVPR 2017 7
Tasks: Image Captioning, Sentence

Visual attention has been successfully applied in structural prediction tasks
such as visual captioning and question answering. Existing visual attention
models are generally spatial, i.e., the attention is modeled as spatial
probabilities that re-weight the last conv-layer feature map of a CNN encoding
an input image. However, we argue that such spatial attention does not
necessarily conform to the attention mechanism --- a dynamic feature extractor
that combines contextual fixations over time, as CNN features are naturally
spatial, channel-wise and multi-layer. In this paper, we introduce a novel
convolutional neural network dubbed SCA-CNN that incorporates Spatial and
Channel-wise Attentions in a CNN. In the task of image captioning, SCA-CNN
dynamically modulates the sentence generation context in multi-layer feature
maps, encoding where (i.e., attentive spatial locations at multiple layers) and
what (i.e., attentive channels) the visual attention is. We evaluate the
proposed SCA-CNN architecture on three benchmark image captioning datasets:
Flickr8K, Flickr30K, and MSCOCO. It is consistently observed that SCA-CNN
significantly outperforms state-of-the-art visual attention-based image
captioning methods.
Sampling-based speech parameter generation using moment-matching networks
Authors: Shinnosuke Takamichi, Tomoki Koriyama, Hiroshi Saruwatari
Date: 2017-04-12 00:00:00
Tasks: Speech Synthesis

This paper presents sampling-based speech parameter generation using
moment-matching networks for Deep Neural Network (DNN)-based speech synthesis.
Although people never produce exactly the same speech even if we try to express
the same linguistic and para-linguistic information, typical statistical speech
synthesis produces completely the same speech, i.e., there is no
inter-utterance variation in synthetic speech. To give synthetic speech natural
inter-utterance variation, this paper builds DNN acoustic models that make it
possible to randomly sample speech parameters. The DNNs are trained so that
they make the moments of generated speech parameters close to those of natural
speech parameters. Since the variation of speech parameters is compressed into
a low-dimensional simple prior noise vector, our algorithm has lower
computation cost than direct sampling of speech parameters. As the first step
towards generating synthetic speech that has natural inter-utterance variation,
this paper investigates whether or not the proposed sampling-based generation
deteriorates synthetic speech quality. In evaluation, we compare speech quality
of conventional maximum likelihood-based generation and proposed sampling-based
generation. The result demonstrates the proposed generation causes no
degradation in speech quality.
Doubly Stochastic Primal-Dual Coordinate Method for Bilinear Saddle-Point Problem
Authors: Adams Wei Yu, Qihang Lin, Tianbao Yang
Date: 2015-08-14 00:00:00

We propose a doubly stochastic primal-dual coordinate optimization algorithm
for empirical risk minimization, which can be formulated as a bilinear
saddle-point problem. In each iteration, our method randomly samples a block of
coordinates of the primal and dual solutions to update. The linear convergence
of our method could be established in terms of 1) the distance from the current
iterate to the optimal solution and 2) the primal-dual objective gap. We show
that the proposed method has a lower overall complexity than existing
coordinate methods when either the data matrix has a factorized structure or
the proximal mapping on each block is computationally expensive, e.g.,
involving an eigenvalue decomposition. The efficiency of the proposed method is
confirmed by empirical studies on several real applications, such as the
multi-task large margin nearest neighbor problem.
Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning
Authors: Matthew Riemer, Elham Khabiri, Richard Goodwin
Date: 2017-04-12 00:00:00
Tasks: Sentiment Analysis, Transfer Learning

Although neural networks are well suited for sequential transfer learning
tasks, the catastrophic forgetting problem hinders proper integration of prior
knowledge. In this work, we propose a solution to this problem by using a
multi-task objective based on the idea of distillation and a mechanism that
directly penalizes forgetting at the shared representation layer during the
knowledge integration phase of training. We demonstrate our approach on a
Twitter domain sentiment analysis task with sequential knowledge transfer from
four related tasks. We show that our technique outperforms networks fine-tuned
to the target task. Additionally, we show both through empirical evidence and
examples that it does not forget useful knowledge from the source task that is
forgotten during standard fine-tuning. Surprisingly, we find that first
distilling a human made rule based sentiment engine into a recurrent neural
network and then integrating the knowledge with the target task data leads to a
substantial gain in generalization performance. Our experiments demonstrate the
power of multi-source transfer techniques in practical text analytics problems
when paired with distillation. In particular, for the SemEval 2016 Task 4
Subtask A (Nakov et al., 2016) dataset we surpass the state of the art
established during the competition with a comparatively simple model
architecture that is not even competitive when trained on only the labeled task
specific data.
Recovery of Sparse and Low Rank Components of Matrices Using Iterative Method with Adaptive Thresholding
Authors: Nematollah Zarmehi, Farokh Marvasti
Date: 2017-03-09 00:00:00

In this letter, we propose an algorithm for recovery of sparse and low rank
components of matrices using an iterative method with adaptive thresholding. In
each iteration, the low rank and sparse components are obtained using a
thresholding operator. This algorithm is fast and can be implemented easily. We
compare it with one of the most common fast methods in which the rank and
sparsity are approximated by $\ell_1$ norm. We also apply it to some real
applications where the noise is not so sparse. The simulation results show that
it has a suitable performance with low run-time.
Learning Features by Watching Objects Move
Authors: Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, Bharath Hariharan
Date: 2016-12-19 00:00:00
Proceeding: CVPR 2017 7
Tasks: object-detection, Object Detection, Transfer Learning

This paper presents a novel yet intuitive approach to unsupervised feature
learning. Inspired by the human visual system, we explore whether low-level
motion-based grouping cues can be used to learn an effective visual
representation. Specifically, we use unsupervised motion-based segmentation on
videos to obtain segments, which we use as 'pseudo ground truth' to train a
convolutional network to segment objects from a single frame. Given the
extensive evidence that motion plays a key role in the development of the human
visual system, we hope that this straightforward approach to unsupervised
learning will be more effective than cleverly designed 'pretext' tasks studied
in the literature. Indeed, our extensive experiments show that this is the
case. When used for transfer learning on object detection, our representation
significantly outperforms previous unsupervised approaches across multiple
settings, especially when training data for the target task is scarce.
Cascaded Face Alignment via Intimacy Definition Feature
Authors: Hailiang Li, Kin-Man Lam, Edmond M. Y. Chiu, Kangheng Wu, Zhibin Lei
Date: 2016-11-21 00:00:00
Tasks: Face Alignment, regression

In this paper, we present a random-forest based fast cascaded regression
model for face alignment, via a novel local feature. Our proposed local
lightweight feature, namely intimacy definition feature (IDF), is more
discriminative than landmark pose-indexed feature, more efficient than
histogram of oriented gradients (HOG) feature and scale-invariant feature
transform (SIFT) feature, and more compact than the local binary feature (LBF).
Experimental results show that our approach achieves state-of-the-art
performance when tested on the most challenging datasets. Compared with an
LBF-based algorithm, our method can achieve about two times the speed-up and
more than 20% improvement, in terms of alignment accuracy measurement, and save
an order of magnitude of memory requirement.
Scene Graph Generation by Iterative Message Passing
Authors: Danfei Xu, Yuke Zhu, Christopher B. Choy, Li Fei-Fei
Date: 2017-01-10 00:00:00
Proceeding: CVPR 2017 7
Tasks: Graph Generation, Panoptic Scene Graph Generation, Scene Graph Generation

Understanding a visual scene goes beyond recognizing individual objects in
isolation. Relationships between objects also constitute rich semantic
information about the scene. In this work, we explicitly model the objects and
their relationships using scene graphs, a visually-grounded graphical structure
of an image. We propose a novel end-to-end model that generates such structured
scene representation from an input image. The model solves the scene graph
inference problem using standard RNNs and learns to iteratively improves its
predictions via message passing. Our joint inference model can take advantage
of contextual cues to make better predictions on objects and their
relationships. The experiments show that our model significantly outperforms
previous methods for generating scene graphs using Visual Genome dataset and
inferring support relations with NYU Depth v2 dataset.
FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos
Authors: Suyog Dutt Jain, Bo Xiong, Kristen Grauman
Date: 2017-01-19 00:00:00
Proceeding: CVPR 2017
Tasks: Segmentation, Structured Prediction, Unsupervised Video Object Segmentation, Video Segmentation, Video Semantic Segmentation

We propose an end-to-end learning framework for segmenting generic objects in
videos. Our method learns to combine appearance and motion information to
produce pixel level segmentation masks for all prominent objects in videos. We
formulate this task as a structured prediction problem and design a two-stream
fully convolutional neural network which fuses together motion and appearance
in a unified framework. Since large-scale video datasets with pixel level
segmentations are problematic, we show how to bootstrap weakly annotated videos
together with existing image recognition datasets for training. Through
experiments on three challenging video segmentation benchmarks, our method
substantially improves the state-of-the-art for segmenting generic (unseen)
objects. Code and pre-trained models are available on the project website.
Material Recognition from Local Appearance in Global Context
Authors: Gabriel Schwartz, Ko Nishino
Date: 2016-11-28 00:00:00
Tasks: Material Recognition

Recognition of materials has proven to be a challenging problem due to the
wide variation in appearance within and between categories. Global image
context, such as where the material is or what object it makes up, can be
crucial to recognizing the material. Existing methods, however, operate on an
implicit fusion of materials and context by using large receptive fields as
input (i.e., large image patches). Many recent material recognition methods
treat materials as yet another set of labels like objects. Materials are,
however, fundamentally different from objects as they have no inherent shape or
defined spatial extent. Approaches that ignore this can only take advantage of
limited implicit context as it appears during training. We instead show that
recognizing materials purely from their local appearance and integrating
separately recognized global contextual cues including objects and places leads
to superior dense, per-pixel, material recognition. We achieve this by training
a fully-convolutional material recognition network end-to-end with only
material category supervision. We integrate object and place estimates to this
network from independent CNNs. This approach avoids the necessity of preparing
an impractically-large amount of training data to cover the product space of
materials, objects, and scenes, while fully leveraging contextual cues for
dense material recognition. Furthermore, we perform a detailed analysis of the
effects of context granularity, spatial resolution, and the network level at
which we introduce context. On a recently introduced comprehensive and diverse
material database \cite{Schwartz2016}, we confirm that our method achieves
state-of-the-art accuracy with significantly less training data compared to
past methods.
Integrating Local Material Recognition with Large-Scale Perceptual Attribute Discovery
Authors: Gabriel Schwartz, Ko Nishino
Date: 2016-04-05 00:00:00
Tasks: Attribute, Material Recognition

Material attributes have been shown to provide a discriminative intermediate
representation for recognizing materials, especially for the challenging task
of recognition from local material appearance (i.e., regardless of object and
scene context). In the past, however, material attributes have been recognized
separately preceding category recognition. In contrast, neuroscience studies on
material perception and computer vision research on object and place
recognition have shown that attributes are produced as a by-product during the
category recognition process. Does the same hold true for material attribute
and category recognition? In this paper, we introduce a novel material category
recognition network architecture to show that perceptual attributes can, in
fact, be automatically discovered inside a local material recognition
framework. The novel material-attribute-category convolutional neural network
(MAC-CNN) produces perceptual material attributes from the intermediate pooling
layers of an end-to-end trained category recognition network using an auxiliary
loss function that encodes human material perception. To train this model, we
introduce a novel large-scale database of local material appearance organized
under a canonical material category taxonomy and careful image patch extraction
that avoids unwanted object and scene context. We show that the discovered
attributes correspond well with semantically-meaningful visual material traits
via Boolean algebra, and enable recognition of previously unseen material
categories given only a few examples. These results have strong implications in
how perceptually meaningful attributes can be learned in other recognition
tasks.
Finding Modes by Probabilistic Hypergraphs Shifting
Authors: Yang Wang, Lin Wu
Date: 2017-04-12 00:00:00
Tasks: Clustering, Graph Matching

In this paper, we develop a novel paradigm, namely hypergraph shift, to find
robust graph modes by probabilistic voting strategy, which are semantically
sound besides the self-cohesiveness requirement in forming graph modes. Unlike
the existing techniques to seek graph modes by shifting vertices based on
pair-wise edges (i.e, an edge with $2$ ends), our paradigm is based on shifting
high-order edges (hyperedges) to deliver graph modes. Specifically, we convert
the problem of seeking graph modes as the problem of seeking maximizers of a
novel objective function with the aim to generate good graph modes based on
sifting edges in hypergraphs. As a result, the generated graph modes based on
dense subhypergraphs may more accurately capture the object semantics besides
the self-cohesiveness requirement. We also formally prove that our technique is
always convergent. Extensive empirical studies on synthetic and real world data
sets are conducted on clustering and graph matching. They demonstrate that our
techniques significantly outperform the existing techniques.
Visual Fashion-Product Search at SK Planet
Authors: Taewan Kim, Seyeong Kim, Sangil Na, Hayoon Kim, Moonki Kim, Byoung-Ki Jeon
Date: 2016-09-26 00:00:00
Tasks: Semantic Similarity, Semantic Textual Similarity

We build a large-scale visual search system which finds similar product
images given a fashion item. Defining similarity among arbitrary
fashion-products is still remains a challenging problem, even there is no exact
ground-truth. To resolve this problem, we define more than 90 fashion-related
attributes, and combination of these attributes can represent thousands of
unique fashion-styles. The fashion-attributes are one of the ingredients to
define semantic similarity among fashion-product images. To build our system at
scale, these fashion-attributes are again used to build an inverted indexing
scheme. In addition to these fashion-attributes for semantic similarity, we
extract colour and appearance features in a region-of-interest (ROI) of a
fashion item for visual similarity. By sharing our approach, we expect active
discussion on that how to apply current computer vision research into the
e-commerce industry.
Variational Gram Functions: Convex Analysis and Optimization
Authors: Amin Jalali, Maryam Fazel, Lin Xiao
Date: 2015-07-16 00:00:00
Tasks: General Classification

We propose a new class of convex penalty functions, called \emph{variational
Gram functions} (VGFs), that can promote pairwise relations, such as
orthogonality, among a set of vectors in a vector space. These functions can
serve as regularizers in convex optimization problems arising from hierarchical
classification, multitask learning, and estimating vectors with disjoint
supports, among other applications. We study convexity for VGFs, and give
efficient characterizations for their convex conjugates, subdifferentials, and
proximal operators. We discuss efficient optimization algorithms for
regularized loss minimization problems where the loss admits a common, yet
simple, variational representation and the regularizer is a VGF. These
algorithms enjoy a simple kernel trick, an efficient line search, as well as
computational advantages over first order methods based on the subdifferential
or proximal maps. We also establish a general representer theorem for such
learning problems. Lastly, numerical experiments on a hierarchical
classification problem are presented to demonstrate the effectiveness of VGFs
and the associated optimization algorithms.
Automatic Discovery, Association Estimation and Learning of Semantic Attributes for a Thousand Categories
Authors: Ziad Al-Halah, Rainer Stiefelhagen
Date: 2017-04-12 00:00:00
Proceeding: CVPR 2017 7
Tasks: Attribute, Zero-Shot Learning

Attribute-based recognition models, due to their impressive performance and
their ability to generalize well on novel categories, have been widely adopted
for many computer vision applications. However, usually both the attribute
vocabulary and the class-attribute associations have to be provided manually by
domain experts or large number of annotators. This is very costly and not
necessarily optimal regarding recognition performance, and most importantly, it
limits the applicability of attribute-based models to large scale data sets. To
tackle this problem, we propose an end-to-end unsupervised attribute learning
approach. We utilize online text corpora to automatically discover a salient
and discriminative vocabulary that correlates well with the human concept of
semantic attributes. Moreover, we propose a deep convolutional model to
optimize class-attribute associations with a linguistic prior that accounts for
noise and missing data in text. In a thorough evaluation on ImageNet, we
demonstrate that our model is able to efficiently discover and learn semantic
attributes at a large scale. Furthermore, we demonstrate that our model
outperforms the state-of-the-art in zero-shot learning on three data sets:
ImageNet, Animals with Attributes and aPascal/aYahoo. Finally, we enable
attribute-based learning on ImageNet and will share the attributes and
associations for future research.
Instance-Level Salient Object Segmentation
Authors: Guanbin Li, Yuan Xie, Liang Lin, Yizhou Yu
Date: 2017-04-12 00:00:00
Proceeding: CVPR 2017 7
Tasks: Instance Segmentation, Object, Saliency Detection, Segmentation, Semantic Segmentation

Image saliency detection has recently witnessed rapid progress due to deep
convolutional neural networks. However, none of the existing methods is able to
identify object instances in the detected salient regions. In this paper, we
present a salient instance segmentation method that produces a saliency mask
with distinct object instance labels for an input image. Our method consists of
three steps, estimating saliency map, detecting salient object contours and
identifying salient object instances. For the first two steps, we propose a
multiscale saliency refinement network, which generates high-quality salient
region masks and salient object contours. Once integrated with multiscale
combinatorial grouping and a MAP-based subset optimization framework, our
method can generate very promising salient object instance segmentation
results. To promote further research and evaluation of salient instance
segmentation, we also construct a new database of 1000 images and their
pixelwise salient instance annotations. Experimental results demonstrate that
our proposed method is capable of achieving state-of-the-art performance on all
public benchmarks for salient region detection as well as on our new dataset
for salient instance segmentation.