pwc-archive/papers-with-abstracts huggingface dataset:
Propositional Knowledge Representation and Reasoning in Restricted Boltzmann Machines
Authors: Son N. Tran
Date: 2017-05-31 00:00:00

While knowledge representation and reasoning are considered the keys for
human-level artificial intelligence, connectionist networks have been shown
successful in a broad range of applications due to their capacity for robust
learning and flexible inference under uncertainty. The idea of representing
symbolic knowledge in connectionist networks has been well-received and
attracted much attention from research community as this can establish a
foundation for integration of scalable learning and sound reasoning. In
previous work, there exist a number of approaches that map logical inference
rules with feed-forward propagation of artificial neural networks (ANN).
However, the discriminative structure of an ANN requires the separation of
input/output variables which makes it difficult for general reasoning where any
variables should be inferable. Other approaches address this issue by employing
generative models such as symmetric connectionist networks, however, they are
difficult and convoluted. In this paper we propose a novel method to represent
propositional formulas in restricted Boltzmann machines which is less complex,
especially in the cases of logical implications and Horn clauses. An
integration system is then developed and evaluated in real datasets which shows
promising results.
Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning
Authors: Wen Sun, J. Andrew Bagnell, Byron Boots
Date: 2018-05-29 00:00:00
Proceeding: ICLR 2018 1
Tasks: Imitation Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In this paper, we propose to combine imitation and reinforcement learning via
the idea of reward shaping using an oracle. We study the effectiveness of the
near-optimal cost-to-go oracle on the planning horizon and demonstrate that the
cost-to-go oracle shortens the learner's planning horizon as function of its
accuracy: a globally optimal oracle can shorten the planning horizon to one,
leading to a one-step greedy Markov Decision Process which is much easier to
optimize, while an oracle that is far away from the optimality requires
planning over a longer horizon to achieve near-optimal performance. Hence our
new insight bridges the gap and interpolates between imitation learning and
reinforcement learning. Motivated by the above mentioned insights, we propose
Truncated HORizon Policy Search (THOR), a method that focuses on searching for
policies that maximize the total reshaped reward over a finite planning horizon
when the oracle is sub-optimal. We experimentally demonstrate that a
gradient-based implementation of THOR can achieve superior performance compared
to RL baselines and IL baselines even when the oracle is sub-optimal.
Fully Convolutional Measurement Network for Compressive Sensing Image Reconstruction
Authors: Jiang Du, Xuemei Xie, Chenye Wang, Guangming Shi, Xun Xu, Yu-Xiang Wang
Date: 2017-11-21 00:00:00
Tasks: Compressive Sensing, Image Reconstruction, SSIM

Recently, deep learning methods have made a significant improvement in
compressive sensing image reconstruction task. In the existing methods, the
scene is measured block by block due to the high computational complexity. This
results in block-effect of the recovered images. In this paper, we propose a
fully convolutional measurement network, where the scene is measured as a
whole. The proposed method powerfully removes the block-effect since the
structure information of scene images is preserved. To make the measure more
flexible, the measurement and the recovery parts are jointly trained. From the
experiments, it is shown that the results by the proposed method outperforms
those by the existing methods in PSNR, SSIM, and visual effect.
Review of Applications of Generalized Regression Neural Networks in Identification and Control of Dynamic Systems
Authors: Ahmad Jobran Al-Mahasneh, Sreenatha G. Anavatti, Matthew A. Garratt
Date: 2018-05-29 00:00:00
Tasks: regression

This paper depicts a brief revision of Generalized Regression Neural Networks
(GRNN) applications in system identification and control of dynamic systems. In
addition, a comparison study between the performance of back-propagation neural
networks and GRNN is presented for system identification problems. The results
of the comparison confirm that GRNN has shorter training time and higher
accuracy than the counterpart back-propagation neural networks.
Fast Convergence for Stochastic and Distributed Gradient Descent in the Interpolation Limit
Authors: Partha P. Mitra
Date: 2018-03-08 00:00:00

Modern supervised learning techniques, particularly those using deep nets,
involve fitting high dimensional labelled data sets with functions containing
very large numbers of parameters. Much of this work is empirical. Interesting
phenomena have been observed that require theoretical explanations; however the
non-convexity of the loss functions complicates the analysis. Recently it has
been proposed that the success of these techniques rests partly in the
effectiveness of the simple stochastic gradient descent algorithm in the so
called interpolation limit in which all labels are fit perfectly. This analysis
is made possible since the SGD algorithm reduces to a stochastic linear system
near the interpolating minimum of the loss function. Here we exploit this
insight by presenting and analyzing a new distributed algorithm for gradient
descent, also in the interpolating limit. The distributed SGD algorithm
presented in the paper corresponds to gradient descent applied to a simple
penalized distributed loss function, $L({\bf w}_1,...,{\bf w}_n) = \Sigma_i
l_i({\bf w}_i) + \mu \sum_{<i,j>}|{\bf w}_i-{\bf w}_j|^2$. Here each node holds
only one sample, and its own parameter vector. The notation $<i,j>$ denotes
edges of a connected graph defining the links between nodes. It is shown that
this distributed algorithm converges linearly (ie the error reduces
exponentially with iteration number), with a rate
$1-\frac{\eta}{n}\lambda_{min}(H)<R<1$ where $\lambda_{min}(H)$ is the smallest
nonzero eigenvalue of the sample covariance or the Hessian H. In contrast with
previous usage of similar penalty functions to enforce consensus between nodes,
in the interpolating limit it is not required to take the penalty parameter to
infinity for consensus to occur. The analysis further reinforces the utility of
the interpolation limit in the theoretical treatment of modern machine learning
algorithms.
Table-to-Text: Describing Table Region with Natural Language
Authors: Junwei Bao, Duyu Tang, Nan Duan, Zhao Yan, Yuanhua Lv, Ming Zhou, Tiejun Zhao
Date: 2018-05-29 00:00:00
Tasks: Language Modeling, Language Modelling, Sentence

In this paper, we present a generative model to generate a natural language
sentence describing a table region, e.g., a row. The model maps a row from a
table to a continuous vector and then generates a natural language sentence by
leveraging the semantics of a table. To deal with rare words appearing in a
table, we develop a flexible copying mechanism that selectively replicates
contents from the table in the output sequence. Extensive experiments
demonstrate the accuracy of the model and the power of the copying mechanism.
On two synthetic datasets, WIKIBIO and SIMPLEQUESTIONS, our model improves the
current state-of-the-art BLEU-4 score from 34.70 to 40.26 and from 33.32 to
39.12, respectively. Furthermore, we introduce an open-domain dataset
WIKITABLETEXT including 13,318 explanatory sentences for 4,962 tables. Our
model achieves a BLEU-4 score of 38.23, which outperforms template based and
language model based approaches.
Retraining-Based Iterative Weight Quantization for Deep Neural Networks
Authors: Dongsoo Lee, Byeongwook Kim
Date: 2018-05-29 00:00:00
Tasks: Model Compression, Quantization

Model compression has gained a lot of attention due to its ability to reduce
hardware resource requirements significantly while maintaining accuracy of
DNNs. Model compression is especially useful for memory-intensive recurrent
neural networks because smaller memory footprint is crucial not only for
reducing storage requirement but also for fast inference operations.
Quantization is known to be an effective model compression method and
researchers are interested in minimizing the number of bits to represent
parameters. In this work, we introduce an iterative technique to apply
quantization, presenting high compression ratio without any modifications to
the training algorithm. In the proposed technique, weight quantization is
followed by retraining the model with full precision weights. We show that
iterative retraining generates new sets of weights which can be quantized with
decreasing quantization loss at each iteration. We also show that quantization
is efficiently able to leverage pruning, another effective model compression
method. Implementation issues on combining the two methods are also addressed.
Our experimental results demonstrate that an LSTM model using 1-bit quantized
weights is sufficient for PTB dataset without any accuracy degradation while
previous methods demand at least 2-4 bits for quantized weights.
Currency exchange prediction using machine learning, genetic algorithms and technical analysis
Authors: Gonçalo Abreu, Rui Neves, Nuno Horta
Date: 2018-05-29 00:00:00
Tasks: BIG-bench Machine Learning, feature selection, Time Series, Time Series Analysis

Technical analysis is used to discover investment opportunities. To test this
hypothesis we propose an hybrid system using machine learning techniques
together with genetic algorithms. Using technical analysis there are more ways
to represent a currency exchange time series than the ones it is possible to
test computationally, i.e., it is unfeasible to search the whole input feature
space thus a genetic algorithm is an alternative. In this work, an architecture
for automatic feature selection is proposed to optimize the cross validated
performance estimation of a Naive Bayes model using a genetic algorithm. The
proposed architecture improves the return on investment of the unoptimized
system from 0,43% to 10,29% in the validation set. The features selected and
the model decision boundary are visualized using the algorithm t-Distributed
Stochastic Neighbor embedding.
Getting to Know Low-light Images with The Exclusively Dark Dataset
Authors: Yuen Peng Loh, Chee Seng Chan
Date: 2018-05-29 00:00:00
Tasks: Image Enhancement, Low-Light Image Enhancement, Object, object-detection, Object Detection

Low-light is an inescapable element of our daily surroundings that greatly
affects the efficiency of our vision. Research works on low-light has seen a
steady growth, particularly in the field of image enhancement, but there is
still a lack of a go-to database as benchmark. Besides, research fields that
may assist us in low-light environments, such as object detection, has glossed
over this aspect even though breakthroughs-after-breakthroughs had been
achieved in recent years, most noticeably from the lack of low-light data (less
than 2% of the total images) in successful public benchmark dataset such as
PASCAL VOC, ImageNet, and Microsoft COCO. Thus, we propose the Exclusively Dark
dataset to elevate this data drought, consisting exclusively of ten different
types of low-light images (i.e. low, ambient, object, single, weak, strong,
screen, window, shadow and twilight) captured in visible light only with image
and object level annotations. Moreover, we share insightful findings in regards
to the effects of low-light on the object detection task by analyzing
visualizations of both hand-crafted and learned features. Most importantly, we
found that the effects of low-light reaches far deeper into the features than
can be solved by simple "illumination invariance'". It is our hope that this
analysis and the Exclusively Dark dataset can encourage the growth in low-light
domain researches on different fields. The Exclusively Dark dataset with its
annotation is available at
https://github.com/cs-chan/Exclusively-Dark-Image-Dataset
Distilling Knowledge for Search-based Structured Prediction
Authors: Yijia Liu, Wanxiang Che, Huaipeng Zhao, Bing Qin, Ting Liu
Date: 2018-05-29 00:00:00
Proceeding: ACL 2018 7
Tasks: Dependency Parsing, Machine Translation, Prediction, Structured Prediction, Transition-Based Dependency Parsing, Translation

Many natural language processing tasks can be modeled into structured
prediction and solved as a search problem. In this paper, we distill an
ensemble of multiple models trained with different initialization into a single
model. In addition to learning to match the ensemble's probability output on
the reference states, we also use the ensemble to explore the search space and
learn from the encountered states in the exploration. Experimental results on
two typical search-based structured prediction tasks -- transition-based
dependency parsing and neural machine translation show that distillation can
effectively improve the single model's performance and the final model achieves
improvements of 1.32 in LAS and 2.65 in BLEU score on these two tasks
respectively over strong baselines and it outperforms the greedy structured
prediction models in previous literatures.
Video Anomaly Detection and Localization via Gaussian Mixture Fully Convolutional Variational Autoencoder
Authors: Yaxiang Fan, Gongjian Wen, Deren Li, Shaohua Qiu, Martin D. Levine
Date: 2018-05-29 00:00:00
Tasks: Anomaly Detection, Decoder, Video Anomaly Detection

We present a novel end-to-end partially supervised deep learning approach for
video anomaly detection and localization using only normal samples. The insight
that motivates this study is that the normal samples can be associated with at
least one Gaussian component of a Gaussian Mixture Model (GMM), while anomalies
either do not belong to any Gaussian component. The method is based on Gaussian
Mixture Variational Autoencoder, which can learn feature representations of the
normal samples as a Gaussian Mixture Model trained using deep learning. A Fully
Convolutional Network (FCN) that does not contain a fully-connected layer is
employed for the encoder-decoder structure to preserve relative spatial
coordinates between the input image and the output feature map. Based on the
joint probabilities of each of the Gaussian mixture components, we introduce a
sample energy based method to score the anomaly of image test patches. A
two-stream network framework is employed to combine the appearance and motion
anomalies, using RGB frames for the former and dynamic flow images, for the
latter. We test our approach on two popular benchmarks (UCSD Dataset and Avenue
Dataset). The experimental results verify the superiority of our method
compared to the state of the arts.
Unsupervised Alignment of Embeddings with Wasserstein Procrustes
Authors: Edouard Grave, Armand Joulin, Quentin Berthet
Date: 2018-05-29 00:00:00
Tasks: Word Embeddings

A library for Multilingual Unsupervised or Supervised word Embeddings
Regularizing deep networks using efficient layerwise adversarial training
Authors: Swami Sankaranarayanan, Arpit Jain, Rama Chellappa, Ser Nam Lim
Date: 2017-05-22 00:00:00

Adversarial training has been shown to regularize deep neural networks in
addition to increasing their robustness to adversarial examples. However, its
impact on very deep state of the art networks has not been fully investigated.
In this paper, we present an efficient approach to perform adversarial training
by perturbing intermediate layer activations and study the use of such
perturbations as a regularizer during training. We use these perturbations to
train very deep models such as ResNets and show improvement in performance both
on adversarial and original test data. Our experiments highlight the benefits
of perturbing intermediate layer activations compared to perturbing only the
inputs. The results on CIFAR-10 and CIFAR-100 datasets show the merits of the
proposed adversarial training approach. Additional results on WideResNets show
that our approach provides significant improvement in classification accuracy
for a given base model, outperforming dropout and other base models of larger
size.
A Neurobiological Evaluation Metric for Neural Network Model Search
Authors: Nathaniel Blanchard, Jeffery Kinnison, Brandon RichardWebster, Pouya Bashivan, Walter J. Scheirer
Date: 2018-05-28 00:00:00
Proceeding: CVPR 2019 6

Neuroscience theory posits that the brain's visual system coarsely identifies
broad object categories via neural activation patterns, with similar objects
producing similar neural responses. Artificial neural networks also have
internal activation behavior in response to stimuli. We hypothesize that
networks exhibiting brain-like activation behavior will demonstrate brain-like
characteristics, e.g., stronger generalization capabilities. In this paper we
introduce a human-model similarity (HMS) metric, which quantifies the
similarity of human fMRI and network activation behavior. To calculate HMS,
representational dissimilarity matrices (RDMs) are created as abstractions of
activation behavior, measured by the correlations of activations to stimulus
pairs. HMS is then the correlation between the fMRI RDM and the neural network
RDM across all stimulus pairs. We test the metric on unsupervised predictive
coding networks, which specifically model visual perception, and assess the
metric for statistical significance over a large range of hyperparameters. Our
experiments show that networks with increased human-model similarity are
correlated with better performance on two computer vision tasks: next frame
prediction and object matching accuracy. Further, HMS identifies networks with
high performance on both tasks. An unexpected secondary finding is that the
metric can be employed during training as an early-stopping mechanism.
Automatic Exposure Compensation for Multi-Exposure Image Fusion
Authors: Yuma Kinoshita, Sayaka Shiota, Hitoshi Kiya
Date: 2018-05-29 00:00:00
Tasks: Multi-Exposure Image Fusion

This paper proposes a novel luminance adjustment method based on automatic
exposure compensation for multi-exposure image fusion. Multi-exposure image
fusion is a method to produce images without saturation regions, by using
photos with different exposures. In conventional works, it has been pointed out
that the quality of those multi-exposure images can be improved by adjusting
the luminance of them. However, how to determine the degree of adjustment has
never been discussed. This paper therefore proposes a way to automatically
determines the degree on the basis of the luminance distribution of input
multi-exposure images. Moreover, new weights, called "simple weights", for
image fusion are also considered for the proposed luminance adjustment method.
Experimental results show that the multi-exposure images adjusted by the
proposed method have better quality than the input multi-exposure ones in terms
of the well-exposedness. It is also confirmed that the proposed simple weights
provide the highest score of statistical naturalness and discrete entropy in
all fusion methods.
Putting a bug in ML: The moth olfactory network learns to read MNIST
Authors: Charles B. Delahunt, J. Nathan Kutz
Date: 2018-02-15 00:00:00
Tasks: BIG-bench Machine Learning, Transfer Learning

We seek to (i) characterize the learning architectures exploited in
biological neural networks for training on very few samples, and (ii) port
these algorithmic structures to a machine learning context. The Moth Olfactory
Network is among the simplest biological neural systems that can learn, and its
architecture includes key structural elements and mechanisms widespread in
biological neural nets, such as cascaded networks, competitive inhibition, high
intrinsic noise, sparsity, reward mechanisms, and Hebbian plasticity. These
structural biological elements, in combination, enable rapid learning.
  MothNet is a computational model of the Moth Olfactory Network, closely
aligned with the moth's known biophysics and with in vivo electrode data
collected from moths learning new odors. We assign this model the task of
learning to read the MNIST digits. We show that MothNet successfully learns to
read given very few training samples (1 to 10 samples per class). In this
few-samples regime, it outperforms standard machine learning methods such as
nearest-neighbors, support-vector machines, and neural networks (NNs), and
matches specialized one-shot transfer-learning methods but without the need for
pre-training. The MothNet architecture illustrates how algorithmic structures
derived from biological brains can be used to build alternative NNs that may
avoid some of the learning rate limitations of current engineered NNs.
A Statistical Recurrent Model on the Manifold of Symmetric Positive Definite Matrices
Authors: Rudrasis Chakraborty, Chun-Hao Yang, Xingjian Zhen, Monami Banerjee, Derek Archer, David Vaillancourt, Vikas Singh, Baba C. Vemuri
Date: 2018-05-29 00:00:00
Proceeding: NeurIPS 2018 12

In a number of disciplines, the data (e.g., graphs, manifolds) to be analyzed
are non-Euclidean in nature. Geometric deep learning corresponds to techniques
that generalize deep neural network models to such non-Euclidean spaces.
Several recent papers have shown how convolutional neural networks (CNNs) can
be extended to learn with graph-based data. In this work, we study the setting
where the data (or measurements) are ordered, longitudinal or temporal in
nature and live on a Riemannian manifold -- this setting is common in a variety
of problems in statistical machine learning, vision and medical imaging. We
show how recurrent statistical recurrent network models can be defined in such
spaces. We give an efficient algorithm and conduct a rigorous analysis of its
statistical properties. We perform extensive numerical experiments
demonstrating competitive performance with state of the art methods but with
significantly less number of parameters. We also show applications to a
statistical analysis task in brain imaging, a regime where deep neural network
models have only been utilized in limited ways.
Leveraged volume sampling for linear regression
Authors: Michał Dereziński, Manfred K. Warmuth, Daniel Hsu
Date: 2018-02-19 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Point Processes, regression

Suppose an $n \times d$ design matrix in a linear regression problem is
given, but the response for each point is hidden unless explicitly requested.
The goal is to sample only a small number $k \ll n$ of the responses, and then
produce a weight vector whose sum of squares loss over all points is at most
$1+\epsilon$ times the minimum. When $k$ is very small (e.g., $k=d$), jointly
sampling diverse subsets of points is crucial. One such method called volume
sampling has a unique and desirable property that the weight vector it produces
is an unbiased estimate of the optimum. It is therefore natural to ask if this
method offers the optimal unbiased estimate in terms of the number of responses
$k$ needed to achieve a $1+\epsilon$ loss approximation.
  Surprisingly we show that volume sampling can have poor behavior when we
require a very accurate approximation -- indeed worse than some i.i.d. sampling
techniques whose estimates are biased, such as leverage score sampling. We then
develop a new rescaled variant of volume sampling that produces an unbiased
estimate which avoids this bad behavior and has at least as good a tail bound
as leverage score sampling: sample size $k=O(d\log d + d/\epsilon)$ suffices to
guarantee total loss at most $1+\epsilon$ times the minimum with high
probability. Thus, we improve on the best previously known sample size for an
unbiased estimator, $k=O(d^2/\epsilon)$.
  Our rescaling procedure leads to a new efficient algorithm for volume
sampling which is based on a determinantal rejection sampling technique with
potentially broader applications to determinantal point processes. Other
contributions include introducing the combinatorics needed for rescaled volume
sampling and developing tail bounds for sums of dependent random matrices which
arise in the process.