pwc-archive/papers-with-abstracts huggingface dataset:
End-to-End Dense Video Captioning with Masked Transformer
Authors: Luowei Zhou, Yingbo Zhou, Jason J. Corso, Richard Socher, Caiming Xiong
Date: 2018-04-03 00:00:00
Proceeding: CVPR 2018 4
Tasks: Decoder, Dense Video Captioning, Video Captioning

Dense video captioning aims to generate text descriptions for all events in
an untrimmed video. This involves both detecting and describing events.
Therefore, all previous methods on dense video captioning tackle this problem
by building two models, i.e. an event proposal and a captioning model, for
these two sub-problems. The models are either trained separately or in
alternation. This prevents direct influence of the language description to the
event proposal, which is important for generating accurate descriptions. To
address this problem, we propose an end-to-end transformer model for dense
video captioning. The encoder encodes the video into appropriate
representations. The proposal decoder decodes from the encoding with different
anchors to form video event proposals. The captioning decoder employs a masking
network to restrict its attention to the proposal event over the encoding
feature. This masking network converts the event proposal to a differentiable
mask, which ensures the consistency between the proposal and captioning during
training. In addition, our model employs a self-attention mechanism, which
enables the use of efficient non-recurrent structure during encoding and leads
to performance improvements. We demonstrate the effectiveness of this
end-to-end model on ActivityNet Captions and YouCookII datasets, where we
achieved 10.12 and 6.58 METEOR score, respectively.
Convolutional Neural Networks Regularized by Correlated Noise
Authors: Shamak Dutta, Bryan Tripp, Graham Taylor
Date: 2018-04-03 00:00:00

Neurons in the visual cortex are correlated in their variability. The
presence of correlation impacts cortical processing because noise cannot be
averaged out over many neurons. In an effort to understand the functional
purpose of correlated variability, we implement and evaluate correlated noise
models in deep convolutional neural networks. Inspired by the cortex,
correlation is defined as a function of the distance between neurons and their
selectivity. We show how to sample from high-dimensional correlated
distributions while keeping the procedure differentiable, so that
back-propagation can proceed as usual. The impact of correlated variability is
evaluated on the classification of occluded and non-occluded images with and
without the presence of other regularization techniques, such as dropout. More
work is needed to understand the effects of correlations in various conditions,
however in 10/12 of the cases we studied, the best performance on occluded
images was obtained from a model with correlated noise.
Patch-based Face Recognition using a Hierarchical Multi-label Matcher
Authors: Lingfeng Zhang, Pengfei Dou, Ioannis A. Kakadiaris
Date: 2018-04-03 00:00:00
Tasks: Face Recognition

This paper proposes a hierarchical multi-label matcher for patch-based face
recognition. In signature generation, a face image is iteratively divided into
multi-level patches. Two different types of patch divisions and signatures are
introduced for 2D facial image and texture-lifted image, respectively. The
matcher training consists of three steps. First, local classifiers are built to
learn the local matching of each patch. Second, the hierarchical relationships
defined between local patches are used to learn the global matching of each
patch. Three ways are introduced to learn the global matching: majority voting,
l1-regularized weighting, and decision rule. Last, the global matchings of
different levels are combined as the final matching. Experimental results on
different face recognition tasks demonstrate the effectiveness of the proposed
matcher at the cost of gallery generalization. Compared with the UR2D system,
the proposed matcher improves the Rank-1 accuracy significantly by 3% and 0.18%
on the UHDB31 dataset and IJB-A dataset, respectively.
StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning
Authors: Kun Shao, Yuanheng Zhu, Dongbin Zhao
Date: 2018-04-03 00:00:00
Tasks: Real-Time Strategy Games, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Starcraft, Transfer Learning

Real-time strategy games have been an important field of game artificial
intelligence in recent years. This paper presents a reinforcement learning and
curriculum transfer learning method to control multiple units in StarCraft
micromanagement. We define an efficient state representation, which breaks down
the complexity caused by the large state space in the game environment. Then a
parameter sharing multi-agent gradientdescent Sarsa({\lambda}) (PS-MAGDS)
algorithm is proposed to train the units. The learning policy is shared among
our units to encourage cooperative behaviors. We use a neural network as a
function approximator to estimate the action-value function, and propose a
reward function to help units balance their move and attack. In addition, a
transfer learning method is used to extend our model to more difficult
scenarios, which accelerates the training process and improves the learning
performance. In small scale scenarios, our units successfully learn to combat
and defeat the built-in AI with 100% win rates. In large scale scenarios,
curriculum transfer learning method is used to progressively train a group of
units, and shows superior performance over some baseline methods in target
scenarios. With reinforcement learning and curriculum transfer learning, our
units are able to learn appropriate strategies in StarCraft micromanagement
scenarios.
In-Bed Pose Estimation: Deep Learning with Shallow Dataset
Authors: Shuangjun Liu, Yu Yin, Sarah Ostadabbas
Date: 2017-11-03 00:00:00
Tasks: Deep Learning, Pose Estimation

Although human pose estimation for various computer vision (CV) applications
has been studied extensively in the last few decades, yet in-bed pose
estimation using camera-based vision methods has been ignored by the CV
community because it is assumed to be identical to the general purpose pose
estimation methods. However, in-bed pose estimation has its own specialized
aspects and comes with specific challenges including the notable differences in
lighting conditions throughout a day and also having different pose
distribution from the common human surveillance viewpoint. In this paper, we
demonstrate that these challenges significantly lessen the effectiveness of
existing general purpose pose estimation models. In order to address the
lighting variation challenge, infrared selective (IRS) image acquisition
technique is proposed to provide uniform quality data under various lighting
conditions. In addition, to deal with unconventional pose perspective, a 2-end
histogram of oriented gradient (HOG) rectification method is presented. In this
work, we explored the idea of employing a pre-trained convolutional neural
network (CNN) model trained on large public datasets of general human poses and
fine-tuning the model using our own shallow in-bed IRS dataset. We developed an
IRS imaging system and collected IRS image data from several realistic
life-size mannequins in a simulated hospital room environment. A pre-trained
CNN called convolutional pose machine (CPM) was repurposed for in-bed pose
estimation by fine-tuning its specific intermediate layers. Using the HOG
rectification method, the pose estimation performance of CPM significantly
improved by 26.4% in PCK0.1 criteria compared to the model without such
rectification.
Contrastive Learning of Emoji-based Representations for Resource-Poor Languages
Authors: Nurendra Choudhary, Rajat Singh, Ishita Bindlish, Manish Shrivastava
Date: 2018-04-03 00:00:00
Tasks: Contrastive Learning

The introduction of emojis (or emoticons) in social media platforms has given
the users an increased potential for expression. We propose a novel method
called Classification of Emojis using Siamese Network Architecture (CESNA) to
learn emoji-based representations of resource-poor languages by jointly
training them with resource-rich languages using a siamese network.
  CESNA model consists of twin Bi-directional Long Short-Term Memory Recurrent
Neural Networks (Bi-LSTM RNN) with shared parameters joined by a contrastive
loss function based on a similarity metric. The model learns the
representations of resource-poor and resource-rich language in a common emoji
space by using a similarity metric based on the emojis present in sentences
from both languages. The model, hence, projects sentences with similar emojis
closer to each other and the sentences with different emojis farther from one
another. Experiments on large-scale Twitter datasets of resource-rich languages
- English and Spanish and resource-poor languages - Hindi and Telugu reveal
that CESNA outperforms the state-of-the-art emoji prediction approaches based
on distributional semantics, semantic rules, lexicon lists and deep neural
network representations without shared parameters.
Sentiment Analysis of Code-Mixed Languages leveraging Resource Rich Languages
Authors: Nurendra Choudhary, Rajat Singh, Ishita Bindlish, Manish Shrivastava
Date: 2018-04-03 00:00:00
Tasks: Clustering, Contrastive Learning, Sentiment Analysis

Code-mixed data is an important challenge of natural language processing
because its characteristics completely vary from the traditional structures of
standard languages.
  In this paper, we propose a novel approach called Sentiment Analysis of
Code-Mixed Text (SACMT) to classify sentences into their corresponding
sentiment - positive, negative or neutral, using contrastive learning. We
utilize the shared parameters of siamese networks to map the sentences of
code-mixed and standard languages to a common sentiment space. Also, we
introduce a basic clustering based preprocessing method to capture variations
of code-mixed transliterated words. Our experiments reveal that SACMT
outperforms the state-of-the-art approaches in sentiment analysis for
code-mixed text by 7.6% in accuracy and 10.1% in F-score.
Emotions are Universal: Learning Sentiment Based Representations of Resource-Poor Languages using Siamese Networks
Authors: Nurendra Choudhary, Rajat Singh, Ishita Bindlish, Manish Shrivastava
Date: 2018-04-03 00:00:00
Tasks: Sentence, Sentiment Analysis

Machine learning approaches in sentiment analysis principally rely on the
abundance of resources. To limit this dependence, we propose a novel method
called Siamese Network Architecture for Sentiment Analysis (SNASA) to learn
representations of resource-poor languages by jointly training them with
resource-rich languages using a siamese network.
  SNASA model consists of twin Bi-directional Long Short-Term Memory Recurrent
Neural Networks (Bi-LSTM RNN) with shared parameters joined by a contrastive
loss function, based on a similarity metric. The model learns the sentence
representations of resource-poor and resource-rich language in a common
sentiment space by using a similarity metric based on their individual
sentiments. The model, hence, projects sentences with similar sentiment closer
to each other and the sentences with different sentiment farther from each
other. Experiments on large-scale datasets of resource-rich languages - English
and Spanish and resource-poor languages - Hindi and Telugu reveal that SNASA
outperforms the state-of-the-art sentiment analysis approaches based on
distributional semantics, semantic rules, lexicon lists and deep neural network
representations without sh
Automatic Normalization of Word Variations in Code-Mixed Social Media Text
Authors: Rajat Singh, Nurendra Choudhary, Manish Shrivastava
Date: 2018-04-03 00:00:00
Tasks: Part-Of-Speech Tagging, POS, POS Tagging, Sentiment Analysis

Social media platforms such as Twitter and Facebook are becoming popular in
multilingual societies. This trend induces portmanteau of South Asian languages
with English. The blend of multiple languages as code-mixed data has recently
become popular in research communities for various NLP tasks. Code-mixed data
consist of anomalies such as grammatical errors and spelling variations. In
this paper, we leverage the contextual property of words where the different
spelling variation of words share similar context in a large noisy social media
text. We capture different variations of words belonging to same context in an
unsupervised manner using distributed representations of words. Our experiments
reveal that preprocessing of the code-mixed dataset based on our approach
improves the performance in state-of-the-art part-of-speech tagging
(POS-tagging) and sentiment analysis tasks.
Low-Shot Learning from Imaginary Data
Authors: Yu-Xiong Wang, Ross Girshick, Martial Hebert, Bharath Hariharan
Date: 2018-01-16 00:00:00
Proceeding: CVPR 2018 6
Tasks: General Classification

Humans can quickly learn new visual concepts, perhaps because they can easily
visualize or imagine what novel objects look like from different views.
Incorporating this ability to hallucinate novel instances of new concepts might
help machine vision systems perform better low-shot learning, i.e., learning
concepts from few examples. We present a novel approach to low-shot learning
that uses this idea. Our approach builds on recent progress in meta-learning
("learning to learn") by combining a meta-learner with a "hallucinator" that
produces additional training examples, and optimizing both models jointly. Our
hallucinator can be incorporated into a variety of meta-learners and provides
significant gains: up to a 6 point boost in classification accuracy when only a
single training example is available, yielding state-of-the-art performance on
the challenging ImageNet low-shot classification benchmark.
Left-Right Comparative Recurrent Model for Stereo Matching
Authors: Zequn Jie, Pengfei Wang, Yonggen Ling, Bo Zhao, Yunchao Wei, Jiashi Feng, Wei Liu
Date: 2018-04-03 00:00:00
Proceeding: CVPR 2018 6
Tasks: Disparity Estimation, model, Stereo Disparity Estimation, Stereo Matching, Stereo Matching Hand

Leveraging the disparity information from both left and right views is
crucial for stereo disparity estimation. Left-right consistency check is an
effective way to enhance the disparity estimation by referring to the
information from the opposite view. However, the conventional left-right
consistency check is an isolated post-processing step and heavily hand-crafted.
This paper proposes a novel left-right comparative recurrent model to perform
left-right consistency checking jointly with disparity estimation. At each
recurrent step, the model produces disparity results for both views, and then
performs online left-right comparison to identify the mismatched regions which
may probably contain erroneously labeled pixels. A soft attention mechanism is
introduced, which employs the learned error maps for better guiding the model
to selectively focus on refining the unreliable regions at the next recurrent
step. In this way, the generated disparity maps are progressively improved by
the proposed recurrent model. Extensive evaluations on KITTI 2015, Scene Flow
and Middlebury benchmarks validate the effectiveness of our model,
demonstrating that state-of-the-art stereo disparity estimation results can be
achieved by this new model.
Estimation of Markov Chain via Rank-Constrained Likelihood
Authors: Xudong Li, Mengdi Wang, Anru Zhang
Date: 2018-04-03 00:00:00
Proceeding: ICML 2018 7

This paper studies the estimation of low-rank Markov chains from empirical
trajectories. We propose a non-convex estimator based on rank-constrained
likelihood maximization. Statistical upper bounds are provided for the
Kullback-Leiber divergence and the $\ell_2$ risk between the estimator and the
true transition matrix. The estimator reveals a compressed state space of the
Markov chain. We also develop a novel DC (difference of convex function)
programming algorithm to tackle the rank-constrained non-smooth optimization
problem. Convergence results are established. Experiments show that the
proposed estimator achieves better empirical performance than other popular
approaches.
Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks
Authors: Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, Tom Goldstein
Date: 2018-04-03 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Data Poisoning, Face Recognition, Transfer Learning

Data poisoning is an attack on machine learning models wherein the attacker
adds examples to the training set to manipulate the behavior of the model at
test time. This paper explores poisoning attacks on neural nets. The proposed
attacks use "clean-labels"; they don't require the attacker to have any control
over the labeling of training data. They are also targeted; they control the
behavior of the classifier on a $\textit{specific}$ test instance without
degrading overall classifier performance. For example, an attacker could add a
seemingly innocuous image (that is properly labeled) to a training set for a
face recognition engine, and control the identity of a chosen person at test
time. Because the attacker does not need to control the labeling function,
poisons could be entered into the training set simply by leaving them on the
web and waiting for them to be scraped by a data collection bot.
  We present an optimization-based method for crafting poisons, and show that
just one single poison image can control classifier behavior when transfer
learning is used. For full end-to-end training, we present a "watermarking"
strategy that makes poisoning reliable using multiple ($\approx$50) poisoned
training instances. We demonstrate our method by generating poisoned frog
images from the CIFAR dataset and using them to manipulate image classifiers.
Multi-Scale Spatially-Asymmetric Recalibration for Image Classification
Authors: Yan Wang, Lingxi Xie, Siyuan Qiao, Ya zhang, Wenjun Zhang, Alan L. Yuille
Date: 2018-04-03 00:00:00
Proceeding: ECCV 2018 9
Tasks: Classification, General Classification, image-classification, Image Classification, Position

Convolution is spatially-symmetric, i.e., the visual features are independent
of its position in the image, which limits its ability to utilize contextual
cues for visual recognition. This paper addresses this issue by introducing a
recalibration process, which refers to the surrounding region of each neuron,
computes an importance value and multiplies it to the original neural response.
Our approach is named multi-scale spatially-asymmetric recalibration (MS-SAR),
which extracts visual cues from surrounding regions at multiple scales, and
designs a weighting scheme which is asymmetric in the spatial domain. MS-SAR is
implemented in an efficient way, so that only small fractions of extra
parameters and computations are required. We apply MS-SAR to several popular
building blocks, including the residual block and the densely-connected block,
and demonstrate its superior performance in both CIFAR and ILSVRC2012
classification tasks.
3D Interpreter Networks for Viewer-Centered Wireframe Modeling
Authors: Jiajun Wu, Tianfan Xue, Joseph J. Lim, Yuandong Tian, Joshua B. Tenenbaum, Antonio Torralba, William T. Freeman
Date: 2018-04-03 00:00:00
Tasks: Image Retrieval, Keypoint Estimation, Object, Retrieval

Understanding 3D object structure from a single image is an important but challenging task in computer vision, mostly due to the lack of 3D object annotations to real images. Previous research tackled this problem by either searching for a 3D shape that best explains 2D annotations, or training purely on synthetic data with ground truth 3D information. In this work, we propose 3D INterpreter Networks (3D-INN), an end-to-end trainable framework that sequentially estimates 2D keypoint heatmaps and 3D object skeletons and poses. Our system learns from both 2D-annotated real images and synthetic 3D data. This is made possible mainly by two technical innovations. First, heatmaps of 2D keypoints serve as an intermediate representation to connect real and synthetic data. 3D-INN is trained on real images to estimate 2D keypoint heatmaps from an input image; it then predicts 3D object structure from heatmaps using knowledge learned from synthetic 3D shapes. By doing so, 3D-INN benefits from the variation and abundance of synthetic 3D objects, without suffering from the domain difference between real and synthesized images, often due to imperfect rendering. Second, we propose a Projection Layer, mapping estimated 3D structure back to 2D. During training, it ensures 3D-INN to predict 3D structure whose projection is consistent with the 2D annotations to real images. Experiments show that the proposed system performs well on both 2D keypoint estimation and 3D structure recovery. We also demonstrate that the recovered 3D information has wide vision applications, such as image retrieval.
Neural Autoregressive Flows
Authors: Chin-wei Huang, David Krueger, Alexandre Lacoste, Aaron Courville
Date: 2018-04-03 00:00:00
Proceeding: ICML 2018 7
Tasks: Density Estimation, Speech Synthesis

Normalizing flows and autoregressive models have been successfully combined
to produce state-of-the-art results in density estimation, via Masked
Autoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-based
speech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows
(IAF). We unify and generalize these approaches, replacing the (conditionally)
affine univariate transformations of MAF/IAF with a more general class of
invertible univariate transformations expressed as monotonic neural networks.
We demonstrate that the proposed neural autoregressive flows (NAF) are
universal approximators for continuous probability distributions, and their
greater expressivity allows them to better capture multimodal target
distributions. Experimentally, NAF yields state-of-the-art performance on a
suite of density estimation tasks and outperforms IAF in variational
autoencoders trained on binarized MNIST.
Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answering
Authors: Duy-Kien Nguyen, Takayuki Okatani
Date: 2018-04-03 00:00:00
Proceeding: CVPR 2018 6
Tasks: Visual Question Answering, Visual Question Answering (VQA)

A key solution to visual question answering (VQA) exists in how to fuse
visual and language features extracted from an input image and question. We
show that an attention mechanism that enables dense, bi-directional
interactions between the two modalities contributes to boost accuracy of
prediction of answers. Specifically, we present a simple architecture that is
fully symmetric between visual and language representations, in which each
question word attends on image regions and each image region attends on
question words. It can be stacked to form a hierarchy for multi-step
interactions between an image-question pair. We show through experiments that
the proposed architecture achieves a new state-of-the-art on VQA and VQA 2.0
despite its small size. We also present qualitative evaluation, demonstrating
how the proposed attention mechanism can generate reasonable attention maps on
images and questions, which leads to the correct answer prediction.
Vanlearning: A Machine Learning SaaS Application for People Without Programming Backgrounds
Authors: Chaochen Wu
Date: 2018-04-03 00:00:00
Tasks: BIG-bench Machine Learning

Although we have tons of machine learning tools to analyze data, most of them
require users have some programming backgrounds. Here we introduce a SaaS
application which allows users analyze their data without any coding and even
without any knowledge of machine learning. Users can upload, train, predict and
download their data by simply clicks their mouses. Our system uses data
pre-processor and validator to relieve the computational cost of our server.
The simple architecture of Vanlearning helps developers can easily maintain and
extend it.