pwc-archive/papers-with-abstracts huggingface dataset:
Universality of Deep Convolutional Neural Networks
Authors: Ding-Xuan Zhou
Date: 2018-05-28 00:00:00
Tasks: Learning Theory, Open-Ended Question Answering, speech-recognition, Speech Recognition

Deep learning has been widely applied and brought breakthroughs in speech
recognition, computer vision, and many other domains. The involved deep neural
network architectures and computational issues have been well studied in
machine learning. But there lacks a theoretical foundation for understanding
the approximation or generalization ability of deep learning methods generated
by the network architectures such as deep convolutional neural networks having
convolutional structures. Here we show that a deep convolutional neural network
(CNN) is universal, meaning that it can be used to approximate any continuous
function to an arbitrary accuracy when the depth of the neural network is large
enough. This answers an open question in learning theory. Our quantitative
estimate, given tightly in terms of the number of free parameters to be
computed, verifies the efficiency of deep CNNs in dealing with large
dimensional data. Our study also demonstrates the role of convolutions in deep
CNNs.
Registration and Fusion of Multi-Spectral Images Using a Novel Edge Descriptor
Authors: Nati Ofir, Shai Silberstein, Dani Rozenbaum, Yosi Keller, Sharon Duvdevani Bar
Date: 2017-11-05 00:00:00
Tasks: Image Registration

In this paper we introduce a fully end-to-end approach for multi-spectral
image registration and fusion. Our method for fusion combines images from
different spectral channels into a single fused image by different approaches
for low and high frequency signals. A prerequisite of fusion is a stage of
geometric alignment between the spectral bands, commonly referred to as
registration. Unfortunately, common methods for image registration of a single
spectral channel do not yield reasonable results on images from different
modalities. For that end, we introduce a new algorithm for multi-spectral image
registration, based on a novel edge descriptor of feature points. Our method
achieves an accurate alignment of a level that allows us to further fuse the
images. As our experiments show, we produce a high quality of multi-spectral
image registration and fusion under many challenging scenarios.
Deep Trustworthy Knowledge Tracing
Authors: Heonseok Ha, Uiwon Hwang, Yongjun Hong, Jahee Jang, Sungroh Yoon
Date: 2018-05-28 00:00:00
Tasks: Knowledge Tracing

Knowledge tracing (KT), a key component of an intelligent tutoring system, is a machine learning technique that estimates the mastery level of a student based on his/her past performance. The objective of KT is to predict a student's response to the next question. Compared with traditional KT models, deep learning-based KT (DLKT) models show better predictive performance because of the representation power of deep neural networks. Various methods have been proposed to improve the performance of DLKT, but few studies have been conducted on the reliability of DLKT. In this work, we claim that the existing DLKTs are not reliable in real education environments. To substantiate the claim, we show limitations of DLKT from various perspectives such as knowledge state update failure, catastrophic forgetting, and non-interpretability. We then propose a novel regularization to address these problems. The proposed method allows us to achieve trustworthy DLKT. In addition, the proposed model which is trained on scenarios with forgetting can also be easily extended to scenarios without forgetting.
Understanding Generalization and Optimization Performance of Deep CNNs
Authors: Pan Zhou, Jiashi Feng
Date: 2018-05-28 00:00:00
Proceeding: ICML 2018 7

This work aims to provide understandings on the remarkable success of deep
convolutional neural networks (CNNs) by theoretically analyzing their
generalization performance and establishing optimization guarantees for
gradient descent based training algorithms. Specifically, for a CNN model
consisting of $l$ convolutional layers and one fully connected layer, we prove
that its generalization error is bounded by
$\mathcal{O}(\sqrt{\dt\widetilde{\varrho}/n})$ where $\theta$ denotes freedom
degree of the network parameters and
$\widetilde{\varrho}=\mathcal{O}(\log(\prod_{i=1}^{l}\rwi{i}
(\ki{i}-\si{i}+1)/p)+\log(\rf))$ encapsulates architecture parameters including
the kernel size $\ki{i}$, stride $\si{i}$, pooling size $p$ and parameter
magnitude $\rwi{i}$. To our best knowledge, this is the first generalization
bound that only depends on $\mathcal{O}(\log(\prod_{i=1}^{l+1}\rwi{i}))$,
tighter than existing ones that all involve an exponential term like
$\mathcal{O}(\prod_{i=1}^{l+1}\rwi{i})$. Besides, we prove that for an
arbitrary gradient descent algorithm, the computed approximate stationary point
by minimizing empirical risk is also an approximate stationary point to the
population risk. This well explains why gradient descent training algorithms
usually perform sufficiently well in practice. Furthermore, we prove the
one-to-one correspondence and convergence guarantees for the non-degenerate
stationary points between the empirical and population risks. It implies that
the computed local minimum for the empirical risk is also close to a local
minimum for the population risk, thus ensuring the good generalization
performance of CNNs.
Improving the Resolution of CNN Feature Maps Efficiently with Multisampling
Authors: Shayan Sadigh, Pradeep Sen
Date: 2018-05-28 00:00:00
Tasks: General Classification, image-classification, Image Classification

We describe a new class of subsampling techniques for CNNs, termed multisampling, that significantly increases the amount of information kept by feature maps through subsampling layers. One version of our method, which we call checkered subsampling, significantly improves the accuracy of state-of-the-art architectures such as DenseNet and ResNet without any additional parameters and, remarkably, improves the accuracy of certain pretrained ImageNet models without any training or fine-tuning. We glean possible insight into the nature of data augmentations and demonstrate experimentally that coarse feature maps are bottlenecking the performance of neural networks in image classification.
Maximum Causal Tsallis Entropy Imitation Learning
Authors: Kyungjae Lee, Sungjoon Choi, Songhwai Oh
Date: 2018-05-22 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Imitation Learning

In this paper, we propose a novel maximum causal Tsallis entropy (MCTE)
framework for imitation learning which can efficiently learn a sparse
multi-modal policy distribution from demonstrations. We provide the full
mathematical analysis of the proposed framework. First, the optimal solution of
an MCTE problem is shown to be a sparsemax distribution, whose supporting set
can be adjusted. The proposed method has advantages over a softmax distribution
in that it can exclude unnecessary actions by assigning zero probability.
Second, we prove that an MCTE problem is equivalent to robust Bayes estimation
in the sense of the Brier score. Third, we propose a maximum causal Tsallis
entropy imitation learning (MCTEIL) algorithm with a sparse mixture density
network (sparse MDN) by modeling mixture weights using a sparsemax
distribution. In particular, we show that the causal Tsallis entropy of an MDN
encourages exploration and efficient mixture utilization while Boltzmann Gibbs
entropy is less effective. We validate the proposed method in two simulation
studies and MCTEIL outperforms existing imitation learning methods in terms of
average returns and learning multi-modal policies.
Clustering by latent dimensions
Authors: Shohei Hidaka, Neeraj Kashyap
Date: 2018-05-28 00:00:00
Tasks: Clustering

This paper introduces a new clustering technique, called {\em dimensional
clustering}, which clusters each data point by its latent {\em pointwise
dimension}, which is a measure of the dimensionality of the data set local to
that point. Pointwise dimension is invariant under a broad class of
transformations. As a result, dimensional clustering can be usefully applied to
a wide range of datasets. Concretely, we present a statistical model which
estimates the pointwise dimension of a dataset around the points in that
dataset using the distance of each point from its $n^{\text{th}}$ nearest
neighbor. We demonstrate the applicability of our technique to the analysis of
dynamical systems, images, and complex human movements.
Dual Policy Iteration
Authors: Wen Sun, Geoffrey J. Gordon, Byron Boots, J. Andrew Bagnell
Date: 2018-05-28 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: continuous-control, Continuous Control

Recently, a novel class of Approximate Policy Iteration (API) algorithms have
demonstrated impressive practical performance (e.g., ExIt from [2],
AlphaGo-Zero from [27]). This new family of algorithms maintains, and
alternately optimizes, two policies: a fast, reactive policy (e.g., a deep
neural network) deployed at test time, and a slow, non-reactive policy (e.g.,
Tree Search), that can plan multiple steps ahead. The reactive policy is
updated under supervision from the non-reactive policy, while the non-reactive
policy is improved with guidance from the reactive policy. In this work we
study this Dual Policy Iteration (DPI) strategy in an alternating optimization
framework and provide a convergence analysis that extends existing API theory.
We also develop a special instance of this framework which reduces the update
of non-reactive policies to model-based optimal control using learned local
models, and provides a theoretically sound way of unifying model-free and
model-based RL approaches with unknown dynamics. We demonstrate the efficacy of
our approach on various continuous control Markov Decision Processes.
Low-Rank Tensor Completion by Truncated Nuclear Norm Regularization
Authors: Shengke Xue, Wenyuan Qiu, Fan Liu, Xinyu Jin
Date: 2017-12-03 00:00:00

Currently, low-rank tensor completion has gained cumulative attention in
recovering incomplete visual data whose partial elements are missing. By taking
a color image or video as a three-dimensional (3D) tensor, previous studies
have suggested several definitions of tensor nuclear norm. However, they have
limitations and may not properly approximate the real rank of a tensor.
Besides, they do not explicitly use the low-rank property in optimization. It
is proved that the recently proposed truncated nuclear norm (TNN) can replace
the traditional nuclear norm, as a better estimation to the rank of a matrix.
Thus, this paper presents a new method called the tensor truncated nuclear norm
(T-TNN), which proposes a new definition of tensor nuclear norm and extends the
truncated nuclear norm from the matrix case to the tensor case. Beneficial from
the low rankness of TNN, our approach improves the efficacy of tensor
completion. We exploit the previously proposed tensor singular value
decomposition and the alternating direction method of multipliers in
optimization. Extensive experiments on real-world videos and images demonstrate
that the performance of our approach is superior to those of existing methods.
Deep Adversarial Context-Aware Landmark Detection for Ultrasound Imaging
Authors: Ahmet Tuysuzoglu, Jeremy Tan, Kareem Eissa, Atilla P. Kiraly, Mamadou Diallo, Ali Kamen
Date: 2018-05-28 00:00:00
Tasks: GPU

Real-time localization of prostate gland in trans-rectal ultrasound images is
a key technology that is required to automate the ultrasound guided prostate
biopsy procedures. In this paper, we propose a new deep learning based approach
which is aimed at localizing several prostate landmarks efficiently and
robustly. We propose a multitask learning approach primarily to make the
overall algorithm more contextually aware. In this approach, we not only
consider the explicit learning of landmark locations, but also build-in a
mechanism to learn the contour of the prostate. This multitask learning is
further coupled with an adversarial arm to promote the generation of feasible
structures. We have trained this network using ~4000 labeled trans-rectal
ultrasound images and tested on an independent set of images with ground truth
landmark locations. We have achieved an overall Dice score of 92.6% for the
adversarially trained multitask approach, which is significantly better than
the Dice score of 88.3% obtained by only learning of landmark locations. The
overall mean distance error using the adversarial multitask approach has also
improved by 20% while reducing the standard deviation of the error compared to
learning landmark locations only. In terms of computational complexity both
approaches can process the images in real-time using standard computer with a
standard CUDA enabled GPU.
Doing the impossible: Why neural networks can be trained at all
Authors: Nathan O. Hodas, Panos Stinis
Date: 2018-05-13 00:00:00
Tasks: All, Protein Folding

As deep neural networks grow in size, from thousands to millions to billions
of weights, the performance of those networks becomes limited by our ability to
accurately train them. A common naive question arises: if we have a system with
billions of degrees of freedom, don't we also need billions of samples to train
it? Of course, the success of deep learning indicates that reliable models can
be learned with reasonable amounts of data. Similar questions arise in protein
folding, spin glasses and biological neural networks. With effectively infinite
potential folding/spin/wiring configurations, how does the system find the
precise arrangement that leads to useful and robust results? Simple sampling of
the possible configurations until an optimal one is reached is not a viable
option even if one waited for the age of the universe. On the contrary, there
appears to be a mechanism in the above phenomena that forces them to achieve
configurations that live on a low-dimensional manifold, avoiding the curse of
dimensionality. In the current work we use the concept of mutual information
between successive layers of a deep neural network to elucidate this mechanism
and suggest possible ways of exploiting it to accelerate training. We show that
adding structure to the neural network that enforces higher mutual information
between layers speeds training and leads to more accurate results. High mutual
information between layers implies that the effective number of free parameters
is exponentially smaller than the raw number of tunable weights.
Perceive Your Users in Depth: Learning Universal User Representations from Multiple E-commerce Tasks
Authors: Yabo Ni, Dan Ou, Shichen Liu, Xiang Li, Wenwu Ou, An-Xiang Zeng, Luo Si
Date: 2018-05-28 00:00:00

Tasks such as search and recommendation have become increas- ingly important
for E-commerce to deal with the information over- load problem. To meet the
diverse needs of di erent users, person- alization plays an important role. In
many large portals such as Taobao and Amazon, there are a bunch of di erent
types of search and recommendation tasks operating simultaneously for person-
alization. However, most of current techniques address each task separately.
This is suboptimal as no information about users shared across di erent tasks.
In this work, we propose to learn universal user representations across
multiple tasks for more e ective personalization. In partic- ular, user
behavior sequences (e.g., click, bookmark or purchase of products) are modeled
by LSTM and attention mechanism by integrating all the corresponding content,
behavior and temporal information. User representations are shared and learned
in an end-to-end setting across multiple tasks. Bene ting from better
information utilization of multiple tasks, the user representations are more e
ective to re ect their interests and are more general to be transferred to new
tasks. We refer this work as Deep User Perception Network (DUPN) and conduct an
extensive set of o ine and online experiments. Across all tested ve di erent
tasks, our DUPN consistently achieves better results by giving more e ective
user representations. Moreover, we deploy DUPN in large scale operational tasks
in Taobao. Detailed implementations, e.g., incre- mental model updating, are
also provided to address the practical issues for the real world applications.
An Efficient and Flexible Spike Train Model via Empirical Bayes
Authors: Qi She, Xiaoli Wu, Beth Jelfs, Adam S. Charles, Rosa H. M. Chan
Date: 2016-05-10 00:00:00
Tasks: Bayesian Inference, parameter estimation

Accurate statistical models of neural spike responses can characterize the information carried by neural populations. But the limited samples of spike counts during recording usually result in model overfitting. Besides, current models assume spike counts to be Poisson-distributed, which ignores the fact that many neurons demonstrate over-dispersed spiking behaviour. Although the Negative Binomial Generalized Linear Model (NB-GLM) provides a powerful tool for modeling over-dispersed spike counts, the maximum likelihood-based standard NB-GLM leads to highly variable and inaccurate parameter estimates. Thus, we propose a hierarchical parametric empirical Bayes method to estimate the neural spike responses among neuronal population. Our method integrates both Generalized Linear Models (GLMs) and empirical Bayes theory, which aims to (1) improve the accuracy and reliability of parameter estimation, compared to the maximum likelihood-based method for NB-GLM and Poisson-GLM; (2) effectively capture the over-dispersion nature of spike counts from both simulated data and experimental data; and (3) provide insight into both neural interactions and spiking behaviours of the neuronal populations. We apply our approach to study both simulated data and experimental neural data. The estimation of simulation data indicates that the new framework can accurately predict mean spike counts simulated from different models and recover the connectivity weights among neural populations. The estimation based on retinal neurons demonstrate the proposed method outperforms both NB-GLM and Poisson-GLM in terms of the predictive log-likelihood of held-out data. Codes are available in https://doi.org/10.5281/zenodo.4704423
Significance testing in non-sparse high-dimensional linear models
Authors: Yinchu Zhu, Jelena Bradic
Date: 2016-10-07 00:00:00
Tasks: Vocal Bursts Intensity Prediction

In high-dimensional linear models, the sparsity assumption is typically made,
stating that most of the parameters are equal to zero. Under the sparsity
assumption, estimation and, recently, inference have been well studied.
However, in practice, sparsity assumption is not checkable and more importantly
is often violated; a large number of covariates might be expected to be
associated with the response, indicating that possibly all, rather than just a
few, parameters are non-zero. A natural example is a genome-wide gene
expression profiling, where all genes are believed to affect a common disease
marker. We show that existing inferential methods are sensitive to the sparsity
assumption, and may, in turn, result in the severe lack of control of Type-I
error. In this article, we propose a new inferential method, named CorrT, which
is robust to model misspecification such as heteroscedasticity and lack of
sparsity. CorrT is shown to have Type I error approaching the nominal level for
\textit{any} models and Type II error approaching zero for sparse and many
dense models.
  In fact, CorrT is also shown to be optimal in a variety of frameworks:
sparse, non-sparse and hybrid models where sparse and dense signals are mixed.
Numerical experiments show a favorable performance of the CorrT test compared
to the state-of-the-art methods.
Designing for Democratization: Introducing Novices to Artificial Intelligence Via Maker Kits
Authors: Victor Dibia, Aaron Cox, Justin Weisz
Date: 2018-05-28 00:00:00

Existing research highlight the myriad of benefits realized when technology
is sufficiently democratized and made accessible to non-technical or novice
users. However, democratizing complex technologies such as artificial
intelligence (AI) remains hard. In this work, we draw on theoretical
underpinnings from the democratization of innovation, in exploring the design
of maker kits that help introduce novice users to complex technologies. We
report on our work designing TJBot: an open source cardboard robot that can be
programmed using pre-built AI services. We highlight principles we adopted in
this process (approachable design, simplicity, extensibility and
accessibility), insights we learned from showing the kit at workshops (66
participants) and how users interacted with the project on GitHub over a
12-month period (Nov 2016 - Nov 2017). We find that the project succeeds in
attracting novice users (40% of users who forked the project are new to GitHub)
and a variety of demographics are interested in prototyping use cases such as
home automation, task delegation, teaching and learning.
Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach
Authors: Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, Shuicheng Yan
Date: 2017-03-24 00:00:00
Proceeding: CVPR 2017 7
Tasks: Classification, General Classification, Object, Segmentation, Semantic Segmentation, Weakly supervised Semantic Segmentation, Weakly-Supervised Semantic Segmentation

We investigate a principle way to progressively mine discriminative object
regions using classification networks to address the weakly-supervised semantic
segmentation problems. Classification networks are only responsive to small and
sparse discriminative regions from the object of interest, which deviates from
the requirement of the segmentation task that needs to localize dense, interior
and integral regions for pixel-wise inference. To mitigate this gap, we propose
a new adversarial erasing approach for localizing and expanding object regions
progressively. Starting with a single small object region, our proposed
approach drives the classification network to sequentially discover new and
complement object regions by erasing the current mined regions in an
adversarial manner. These localized regions eventually constitute a dense and
complete object region for learning semantic segmentation. To further enhance
the quality of the discovered regions by adversarial erasing, an online
prohibitive segmentation learning approach is developed to collaborate with
adversarial erasing by providing auxiliary segmentation supervision modulated
by the more reliable classification scores. Despite its apparent simplicity,
the proposed approach achieves 55.0% and 55.7% mean Intersection-over-Union
(mIoU) scores on PASCAL VOC 2012 val and test sets, which are the new
state-of-the-arts.