pwc-archive/papers-with-abstracts huggingface dataset:
How to Make a BLT Sandwich? Learning to Reason towards Understanding Web Instructional Videos
Authors: Shaojie Wang, Wentian Zhao, Ziyi Kou, Chenliang Xu
Date: 2018-12-02 00:00:00
Tasks: Logical Reasoning, Question Answering, Video Understanding

Understanding web instructional videos is an essential branch of video
understanding in two aspects. First, most existing video methods focus on
short-term actions for a-few-second-long video clips; these methods are not
directly applicable to long videos. Second, unlike unconstrained long videos,
e.g., movies, instructional videos are more structured in that they have
step-by-step procedure constraining the understanding task. In this paper, we
study reasoning on instructional videos via question-answering (QA).
Surprisingly, it has not been an emphasis in the video community despite its
rich applications. We thereby introduce YouQuek, an annotated QA dataset for
instructional videos based on the recent YouCook2. The questions in YouQuek are
not limited to cues on one frame but related to logical reasoning in the
temporal dimension. Observing the lack of effective representations for
modeling long videos, we propose a set of carefully designed models including a
novel Recurrent Graph Convolutional Network (RGCN) that captures both temporal
order and relation information. Furthermore, we study multiple modalities
including description and transcripts for the purpose of boosting video
understanding. Extensive experiments on YouQuek suggest that RGCN performs the
best in terms of QA accuracy and a better performance is gained by introducing
human annotated description.
Analysis on Gradient Propagation in Batch Normalized Residual Networks
Authors: Abhishek Panigrahi, Yueru Chen, C. -C. Jay Kuo
Date: 2018-12-02 00:00:00
Proceeding: ICLR 2018 1

We conduct mathematical analysis on the effect of batch normalization (BN) on
gradient backpropogation in residual network training, which is believed to
play a critical role in addressing the gradient vanishing/explosion problem, in
this work. By analyzing the mean and variance behavior of the input and the
gradient in the forward and backward passes through the BN and residual
branches, respectively, we show that they work together to confine the gradient
variance to a certain range across residual blocks in backpropagation. As a
result, the gradient vanishing/explosion problem is avoided. We also show the
relative importance of batch normalization w.r.t. the residual branches in
residual networks.
Regularized Wasserstein Means for Aligning Distributional Data
Authors: Liang Mi, Wen Zhang, Yalin Wang
Date: 2018-12-02 00:00:00
Tasks: Domain Adaptation

We propose to align distributional data from the perspective of Wasserstein means. We raise the problem of regularizing Wasserstein means and propose several terms tailored to tackle different problems. Our formulation is based on the variational transportation to distribute a sparse discrete measure into the target domain. The resulting sparse representation well captures the desired property of the domain while reducing the mapping cost. We demonstrate the scalability and robustness of our method with examples in domain adaptation, point set registration, and skeleton layout.
Macro action selection with deep reinforcement learning in StarCraft
Authors: Sijia Xu, Hongyu Kuang, Zhi Zhuang, Renjie Hu, Yang Liu, Huyang Sun
Date: 2018-12-02 00:00:00
Tasks: Deep Reinforcement Learning, Real-Time Strategy Games, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Starcraft

StarCraft (SC) is one of the most popular and successful Real Time Strategy (RTS) games. In recent years, SC is also widely accepted as a challenging testbed for AI research because of its enormous state space, partially observed information, multi-agent collaboration, and so on. With the help of annual AIIDE and CIG competitions, a growing number of SC bots are proposed and continuously improved. However, a large gap remains between the top-level bot and the professional human player. One vital reason is that current SC bots mainly rely on predefined rules to select macro actions during their games. These rules are not scalable and efficient enough to cope with the enormous yet partially observed state space in the game. In this paper, we propose a deep reinforcement learning (DRL) framework to improve the selection of macro actions. Our framework is based on the combination of the Ape-X DQN and the Long-Short-Term-Memory (LSTM). We use this framework to build our bot, named as LastOrder. Our evaluation, based on training against all bots from the AIIDE 2017 StarCraft AI competition set, shows that LastOrder achieves an 83% winning rate, outperforming 26 bots in total 28 entrants.
GAN-EM: GAN based EM learning framework
Authors: Wentian Zhao, Shaojie Wang, Zhihuai Xie, Jing Shi, Chenliang Xu
Date: 2018-12-02 00:00:00
Tasks: Clustering, Dimensionality Reduction, General Classification, Image Clustering

Expectation maximization (EM) algorithm is to find maximum likelihood
solution for models having latent variables. A typical example is Gaussian
Mixture Model (GMM) which requires Gaussian assumption, however, natural images
are highly non-Gaussian so that GMM cannot be applied to perform clustering
task on pixel space. To overcome such limitation, we propose a GAN based EM
learning framework that can maximize the likelihood of images and estimate the
latent variables with only the constraint of L-Lipschitz continuity. We call
this model GAN-EM, which is a framework for image clustering, semi-supervised
classification and dimensionality reduction. In M-step, we design a novel loss
function for discriminator of GAN to perform maximum likelihood estimation
(MLE) on data with soft class label assignments. Specifically, a conditional
generator captures data distribution for $K$ classes, and a discriminator tells
whether a sample is real or fake for each class. Since our model is
unsupervised, the class label of real data is regarded as latent variable,
which is estimated by an additional network (E-net) in E-step. The proposed
GAN-EM achieves state-of-the-art clustering and semi-supervised classification
results on MNIST, SVHN and CelebA, as well as comparable quality of generated
images to other recently developed generative models.
Image Score: How to Select Useful Samples
Authors: Simiao Zuo, Jialin Wu
Date: 2018-12-02 00:00:00
Proceeding: ICLR 2019 5
Tasks: Decision Making

There has long been debates on how we could interpret neural networks and
understand the decisions our models make. Specifically, why deep neural
networks tend to be error-prone when dealing with samples that output low
softmax scores. We present an efficient approach to measure the confidence of
decision-making steps by statistically investigating each unit's contribution
to that decision. Instead of focusing on how the models react on datasets, we
study the datasets themselves given a pre-trained model. Our approach is
capable of assigning a score to each sample within a dataset that measures the
frequency of occurrence of that sample's chain of activation. We demonstrate
with experiments that our method could select useful samples to improve deep
neural networks in a semi-supervised leaning setting.
PVRNet: Point-View Relation Neural Network for 3D Shape Recognition
Authors: Haoxuan You, Yifan Feng, Xibin Zhao, Changqing Zou, Rongrong Ji, Yue Gao
Date: 2018-12-02 00:00:00
Tasks: 3D Shape Classification, 3D Shape Recognition, Relation, Relation Network, Retrieval

Three-dimensional (3D) shape recognition has drawn much research attention in
the field of computer vision. The advances of deep learning encourage various
deep models for 3D feature representation. For point cloud and multi-view data,
two popular 3D data modalities, different models are proposed with remarkable
performance. However the relation between point cloud and views has been rarely
investigated. In this paper, we introduce Point-View Relation Network (PVRNet),
an effective network designed to well fuse the view features and the point
cloud feature with a proposed relation score module. More specifically, based
on the relation score module, the point-single-view fusion feature is first
extracted by fusing the point cloud feature and each single view feature with
point-singe-view relation, then the point-multi-view fusion feature is
extracted by fusing the point cloud feature and the features of different
number of views with point-multi-view relation. Finally, the point-single-view
fusion feature and point-multi-view fusion feature are further combined
together to achieve a unified representation for a 3D shape. Our proposed
PVRNet has been evaluated on ModelNet40 dataset for 3D shape classification and
retrieval. Experimental results indicate our model can achieve significant
performance improvement compared with the state-of-the-art models.
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
Authors: Han Cai, Ligeng Zhu, Song Han
Date: 2018-12-02 00:00:00
Proceeding: ICLR 2019 5
Tasks: GPU, Image Classification, Neural Architecture Search

Neural architecture search (NAS) has a great impact by automatically
designing effective neural network architectures. However, the prohibitive
computational demand of conventional NAS algorithms (e.g. $10^4$ GPU hours)
makes it difficult to \emph{directly} search the architectures on large-scale
tasks (e.g. ImageNet). Differentiable NAS can reduce the cost of GPU hours via
a continuous representation of network architecture but suffers from the high
GPU memory consumption issue (grow linearly w.r.t. candidate set size). As a
result, they need to utilize~\emph{proxy} tasks, such as training on a smaller
dataset, or learning with only a few blocks, or training just for a few epochs.
These architectures optimized on proxy tasks are not guaranteed to be optimal
on the target task. In this paper, we present \emph{ProxylessNAS} that can
\emph{directly} learn the architectures for large-scale target tasks and target
hardware platforms. We address the high memory consumption issue of
differentiable NAS and reduce the computational cost (GPU hours and GPU memory)
to the same level of regular training while still allowing a large candidate
set. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of
directness and specialization. On CIFAR-10, our model achieves 2.08\% test
error with only 5.7M parameters, better than the previous state-of-the-art
architecture AmoebaNet-B, while using 6$\times$ fewer parameters. On ImageNet,
our model achieves 3.1\% better top-1 accuracy than MobileNetV2, while being
1.2$\times$ faster with measured GPU latency. We also apply ProxylessNAS to
specialize neural architectures for hardware with direct hardware metrics (e.g.
latency) and provide insights for efficient CNN architecture design.
Iterative Reorganization with Weak Spatial Constraints: Solving Arbitrary Jigsaw Puzzles for Unsupervised Representation Learning
Authors: Chen Wei, Lingxi Xie, Xutong Ren, Yingda Xia, Chi Su, Jiaying Liu, Qi Tian, Alan L. Yuille
Date: 2018-12-02 00:00:00
Proceeding: CVPR 2019 6
Tasks: General Classification, image-classification, Image Classification, object-detection, Object Detection, Representation Learning, Semantic Segmentation

Learning visual features from unlabeled image data is an important yet
challenging task, which is often achieved by training a model on some
annotation-free information. We consider spatial contexts, for which we solve
so-called jigsaw puzzles, i.e., each image is cut into grids and then
disordered, and the goal is to recover the correct configuration. Existing
approaches formulated it as a classification task by defining a fixed mapping
from a small subset of configurations to a class set, but these approaches
ignore the underlying relationship between different configurations and also
limit their application to more complex scenarios. This paper presents a novel
approach which applies to jigsaw puzzles with an arbitrary grid size and
dimensionality. We provide a fundamental and generalized principle, that weaker
cues are easier to be learned in an unsupervised manner and also transfer
better. In the context of puzzle recognition, we use an iterative manner which,
instead of solving the puzzle all at once, adjusts the order of the patches in
each step until convergence. In each step, we combine both unary and binary
features on each patch into a cost function judging the correctness of the
current configuration. Our approach, by taking similarity between puzzles into
consideration, enjoys a more reasonable way of learning visual knowledge. We
verify the effectiveness of our approach in two aspects. First, it is able to
solve arbitrarily complex puzzles, including high-dimensional puzzles, that
prior methods are difficult to handle. Second, it serves as a reliable way of
network initialization, which leads to better transfer performance in a few
visual recognition tasks including image classification, object detection, and
semantic segmentation.
End-to-end Learning of Convolutional Neural Net and Dynamic Programming for Left Ventricle Segmentation
Authors: Nhat M. Nguyen, Nilanjan Ray
Date: 2018-12-02 00:00:00
Proceeding: MIDL 2019 7
Tasks: Left Ventricle Segmentation

Differentiable programming is able to combine different functions or programs in a processing pipeline with the goal of applying end-to-end learning or optimization. A significant impediment is the non-differentiable nature of some algorithms. We propose to use synthetic gradients (SG) to overcome this difficulty. SG uses the universal function approximation property of neural networks. We apply SG to combine convolutional neural network (CNN) with dynamic programming (DP) in end-to-end learning for segmenting left ventricle from short axis view of heart MRI. Our experiments show that end-to-end combination of CNN and DP requires fewer labeled images to achieve a significantly better segmentation accuracy than using only CNN.
CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark
Authors: Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, Cewu Lu
Date: 2018-12-02 00:00:00
Proceeding: CVPR 2019 6
Tasks: Keypoint Detection, Multi-Person Pose Estimation, Pose Estimation

Multi-person pose estimation is fundamental to many computer vision tasks and
has made significant progress in recent years. However, few previous methods
explored the problem of pose estimation in crowded scenes while it remains
challenging and inevitable in many scenarios. Moreover, current benchmarks
cannot provide an appropriate evaluation for such cases. In this paper, we
propose a novel and efficient method to tackle the problem of pose estimation
in the crowd and a new dataset to better evaluate algorithms. Our model
consists of two key components: joint-candidate single person pose estimation
(SPPE) and global maximum joints association. With multi-peak prediction for
each joint and global association using graph model, our method is robust to
inevitable interference in crowded scenes and very efficient in inference. The
proposed method surpasses the state-of-the-art methods on CrowdPose dataset by
5.2 mAP and results on MSCOCO dataset demonstrate the generalization ability of
our method. Source code and dataset will be made publicly available.
Fake News: A Survey of Research, Detection Methods, and Opportunities
Authors: Xinyi Zhou, Reza Zafarani
Date: 2018-12-02 00:00:00
Tasks: Survey

The explosive growth in fake news and its erosion to democracy, justice, and
public trust has increased the demand for fake news analysis, detection and
intervention. This survey comprehensively and systematically reviews fake news
research. The survey identifies and specifies fundamental theories across
various disciplines, e.g., psychology and social science, to facilitate and
enhance the interdisciplinary research of fake news. Current fake news research
is reviewed, summarized and evaluated. These studies focus on fake news from
four perspective: (1) the false knowledge it carries, (2) its writing style,
(3) its propagation patterns, and (4) the credibility of its creators and
spreaders. We characterize each perspective with various analyzable and
utilizable information provided by news and its spreaders, various strategies
and frameworks that are adaptable, and techniques that are applicable. By
reviewing the characteristics of fake news and open issues in fake news
studies, we highlight some potential research tasks at the end of this survey.
ECO: Egocentric Cognitive Mapping
Authors: Jayant Sharma, Zixing Wang, Alberto Speranzon, Vijay Venkataraman, Hyun Soo Park
Date: 2018-12-02 00:00:00
Tasks: Domain Adaptation, Navigate

We present a new method to localize a camera within a previously unseen
environment perceived from an egocentric point of view. Although this is, in
general, an ill-posed problem, humans can effortlessly and efficiently
determine their relative location and orientation and navigate into a
previously unseen environments, e.g., finding a specific item in a new grocery
store. To enable such a capability, we design a new egocentric representation,
which we call ECO (Egocentric COgnitive map). ECO is biologically inspired, by
the cognitive map that allows human navigation, and it encodes the surrounding
visual semantics with respect to both distance and orientation. ECO possesses
three main properties: (1) reconfigurability: complex semantics and geometry is
captured via the synthesis of atomic visual representations (e.g., image
patch); (2) robustness: the visual semantics are registered in a geometrically
consistent way (e.g., aligning with respect to the gravity vector,
frontalizing, and rescaling to canonical depth), thus enabling us to learn
meaningful atomic representations; (3) adaptability: a domain adaptation
framework is designed to generalize the learned representation without manual
calibration. As a proof-of-concept, we use ECO to localize a camera within
real-world scenes---various grocery stores---and demonstrate performance
improvements when compared to existing semantic localization approaches.
On variation of gradients of deep neural networks
Authors: Yongdai Kim, Dongha Kim
Date: 2018-12-02 00:00:00

We provide a theoretical explanation of the role of the number of nodes at
each layer in deep neural networks. We prove that the largest variation of a
deep neural network with ReLU activation function arises when the layer with
the fewest nodes changes its activation pattern. An important implication is
that deep neural network is a useful tool to generate functions most of whose
variations are concentrated on a smaller area of the input space near the
boundaries corresponding to the layer with the fewest nodes. In turn, this
property makes the function more invariant to input transformation. That is,
our theoretical result gives a clue about how to design the architecture of a
deep neural network to increase complexity and transformation invariancy
simultaneously.
Multi-modal Capsule Routing for Actor and Action Video Segmentation Conditioned on Natural Language Queries
Authors: Bruce McIntosh, Kevin Duarte, Yogesh S Rawat, Mubarak Shah
Date: 2018-12-02 00:00:00
Tasks: Action Localization, Natural Language Queries, Video Segmentation, Video Semantic Segmentation

In this paper, we propose an end-to-end capsule network for pixel level
localization of actors and actions present in a video. The localization is
performed based on a natural language query through which an actor and action
are specified. We propose to encode both the video as well as textual input in
the form of capsules, which provide more effective representation in comparison
with standard convolution based features. We introduce a novel capsule based
attention mechanism for fusion of video and text capsules for text selected
video segmentation. The attention mechanism is performed via joint EM routing
over video and text capsules for text selected actor and action localization.
The existing works on actor-action localization are mainly focused on
localization in a single frame instead of the full video. Different from
existing works, we propose to perform the localization on all frames of the
video. To validate the potential of the proposed network for actor and action
localization on all the frames of a video, we extend an existing actor-action
dataset (A2D) with annotations for all the frames. The experimental evaluation
demonstrates the effectiveness of the proposed capsule network for text
selective actor and action localization in videos, and it also improves upon
the performance of the existing state-of-the art works on single frame-based
localization.
Plan-Recognition-Driven Attention Modeling for Visual Recognition
Authors: Yantian Zha, Yikang Li, Tianshu Yu, Subbarao Kambhampati, Baoxin Li
Date: 2018-12-02 00:00:00

Human visual recognition of activities or external agents involves an interplay between high-level plan recognition and low-level perception. Given that, a natural question to ask is: can low-level perception be improved by high-level plan recognition? We formulate the problem of leveraging recognized plans to generate better top-down attention maps \cite{gazzaniga2009,baluch2011} to improve the perception performance. We call these top-down attention maps specifically as plan-recognition-driven attention maps. To address this problem, we introduce the Pixel Dynamics Network. Pixel Dynamics Network serves as an observation model, which predicts next states of object points at each pixel location given observation of pixels and pixel-level action feature. This is like internally learning a pixel-level dynamics model. Pixel Dynamics Network is a kind of Convolutional Neural Network (ConvNet), with specially-designed architecture. Therefore, Pixel Dynamics Network could take the advantage of parallel computation of ConvNets, while learning the pixel-level dynamics model. We further prove the equivalence between Pixel Dynamics Network as an observation model, and the belief update in partially observable Markov decision process (POMDP) framework. We evaluate our Pixel Dynamics Network in event recognition tasks. We build an event recognition system, ER-PRN, which takes Pixel Dynamics Network as a subroutine, to recognize events based on observations augmented by plan-recognition-driven attention.
In-silico Risk Analysis of Personalized Artificial Pancreas Controllers via Rare-event Simulation
Authors: Matthew O'Kelly, Aman Sinha, Justin Norden, Hongseok Namkoong
Date: 2018-12-02 00:00:00

Modern treatments for Type 1 diabetes (T1D) use devices known as artificial
pancreata (APs), which combine an insulin pump with a continuous glucose
monitor (CGM) operating in a closed-loop manner to control blood glucose
levels. In practice, poor performance of APs (frequent hyper- or hypoglycemic
events) is common enough at a population level that many T1D patients modify
the algorithms on existing AP systems with unregulated open-source software.
Anecdotally, the patients in this group have shown superior outcomes compared
with standard of care, yet we do not understand how safe any AP system is since
adverse outcomes are rare. In this paper, we construct generative models of
individual patients' physiological characteristics and eating behaviors. We
then couple these models with a T1D simulator approved for pre-clinical trials
by the FDA. Given the ability to simulate patient outcomes in-silico, we
utilize techniques from rare-event simulation theory in order to efficiently
quantify the performance of a device with respect to a particular patient. We
show a 72,000$\times$ speedup in simulation speed over real-time and up to 2-10
times increase in the frequency which we are able to sample adverse conditions
relative to standard Monte Carlo sampling. In practice our toolchain enables
estimates of the likelihood of hypoglycemic events with approximately an order
of magnitude fewer simulations.