pwc-archive/papers-with-abstracts huggingface dataset:
An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family
Authors: Alexandre de Brébisson, Pascal Vincent
Date: 2015-11-16 00:00:00
Tasks: Language Modeling, Language Modelling, Multi-class Classification

In a multi-class classification problem, it is standard to model the output
of a neural network as a categorical distribution conditioned on the inputs.
The output must therefore be positive and sum to one, which is traditionally
enforced by a softmax. This probabilistic mapping allows to use the maximum
likelihood principle, which leads to the well-known log-softmax loss. However
the choice of the softmax function seems somehow arbitrary as there are many
other possible normalizing functions. It is thus unclear why the log-softmax
loss would perform better than other loss alternatives. In particular Vincent
et al. (2015) recently introduced a class of loss functions, called the
spherical family, for which there exists an efficient algorithm to compute the
updates of the output weights irrespective of the output size. In this paper,
we explore several loss functions from this family as possible alternatives to
the traditional log-softmax. In particular, we focus our investigation on
spherical bounds of the log-softmax loss and on two spherical log-likelihood
losses, namely the log-Spherical Softmax suggested by Vincent et al. (2015) and
the log-Taylor Softmax that we introduce. Although these alternatives do not
yield as good results as the log-softmax loss on two language modeling tasks,
they surprisingly outperform it in our experiments on MNIST and CIFAR-10,
suggesting that they might be relevant in a broad range of applications.
On the entropy numbers of the mixed smoothness function classes
Authors: V. Temlyakov
Date: 2016-02-28 00:00:00

Behavior of the entropy numbers of classes of multivariate functions with
mixed smoothness is studied here. This problem has a long history and some
fundamental problems in the area are still open. The main goal of this paper is
to develop a new method of proving the upper bounds for the entropy numbers.
This method is based on recent developments of nonlinear approximation, in
particular, on greedy approximation. This method consists of the following two
steps strategy. At the first step we obtain bounds of the best m-term
approximations with respect to a dictionary. At the second step we use general
inequalities relating the entropy numbers to the best m-term approximations.
For the lower bounds we use the volume estimates method, which is a well known
powerful method for proving the lower bounds for the entropy numbers. It was
used in a number of previous papers.
Bilingual Distributed Word Representations from Document-Aligned Comparable Data
Authors: Ivan Vulić, Marie-Francine Moens
Date: 2015-09-24 00:00:00
Tasks: Representation Learning, Sentence, Translation, Word Embeddings

We propose a new model for learning bilingual word representations from
non-parallel document-aligned data. Following the recent advances in word
representation learning, our model learns dense real-valued word vectors, that
is, bilingual word embeddings (BWEs). Unlike prior work on inducing BWEs which
heavily relied on parallel sentence-aligned corpora and/or readily available
translation resources such as dictionaries, the article reveals that BWEs may
be learned solely on the basis of document-aligned comparable data without any
additional lexical resources nor syntactic information. We present a comparison
of our approach with previous state-of-the-art models for learning bilingual
word representations from comparable data that rely on the framework of
multilingual probabilistic topic modeling (MuPTM), as well as with
distributional local context-counting models. We demonstrate the utility of the
induced BWEs in two semantic tasks: (1) bilingual lexicon extraction, (2)
suggesting word translations in context for polysemous words. Our simple yet
effective BWE-based models significantly outperform the MuPTM-based and
context-counting representation models from comparable data as well as prior
BWE-based models, and acquire the best reported results on both tasks for all
three tested language pairs.
Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping
Authors: Michael Xie, Neal Jean, Marshall Burke, David Lobell, Stefano Ermon
Date: 2015-10-01 00:00:00
Tasks: Humanitarian, Transfer Learning

The lack of reliable data in developing countries is a major obstacle to
sustainable development, food security, and disaster relief. Poverty data, for
example, is typically scarce, sparse in coverage, and labor-intensive to
obtain. Remote sensing data such as high-resolution satellite imagery, on the
other hand, is becoming increasingly available and inexpensive. Unfortunately,
such data is highly unstructured and currently no techniques exist to
automatically extract useful insights to inform policy decisions and help
direct humanitarian efforts. We propose a novel machine learning approach to
extract large-scale socioeconomic indicators from high-resolution satellite
imagery. The main challenge is that training data is very scarce, making it
difficult to apply modern techniques such as Convolutional Neural Networks
(CNN). We therefore propose a transfer learning approach where nighttime light
intensities are used as a data-rich proxy. We train a fully convolutional CNN
model to predict nighttime lights from daytime imagery, simultaneously learning
features that are useful for poverty prediction. The model learns filters
identifying different terrains and man-made structures, including roads,
buildings, and farmlands, without any supervision beyond nighttime lights. We
demonstrate that these learned features are highly informative for poverty
mapping, even approaching the predictive performance of survey data collected
in the field.
A Bayesian baseline for belief in uncommon events
Authors: V. Palonen
Date: 2016-02-25 00:00:00

The plausibility of uncommon events and miracles based on testimony of such
an event has been much discussed. When analyzing the probabilities involved, it
has mostly been assumed that the common events can be taken as data in the
calculations. However, we usually have only testimonies for the common events.
While this difference does not have a significant effect on the inductive part
of the inference, it has a large influence on how one should view the
reliability of testimonies. In this work, a full Bayesian solution is given for
the more realistic case, where one has a large number of testimonies for a
common event and one testimony for an uncommon event. It is seen that, in order
for there to be a large amount of testimonies for a common event, the
testimonies will probably be quite reliable. For this reason, because the
testimonies are quite reliable based on the testimonies for the common events,
the probability for the uncommon event, given a testimony for it, is also
higher. Hence, one should be more open-minded when considering the plausibility
of uncommon events.
Illustrating a neural model of logic computations: The case of Sherlock Holmes' old maxim
Authors: Eduardo Mizraji
Date: 2012-10-28 00:00:00

Natural languages can express some logical propositions that humans are able
to understand. We illustrate this fact with a famous text that Conan Doyle
attributed to Holmes: 'It is an old maxim of mine that when you have excluded
the impossible, whatever remains, however improbable, must be the truth'. This
is a subtle logical statement usually felt as an evident truth. The problem we
are trying to solve is the cognitive reason for such a feeling. We postulate
here that we accept Holmes' maxim as true because our adult brains are equipped
with neural modules that naturally perform modal logical computations.
A 'Gibbs-Newton' Technique for Enhanced Inference of Multivariate Polya Parameters and Topic Models
Authors: Osama Khalifa, David Wolfe Corne, Mike Chantler
Date: 2015-10-22 00:00:00
Tasks: Binary Classification, Topic Models

Hyper-parameters play a major role in the learning and inference process of
latent Dirichlet allocation (LDA). In order to begin the LDA latent variables
learning process, these hyper-parameters values need to be pre-determined. We
propose an extension for LDA that we call 'Latent Dirichlet allocation Gibbs
Newton' (LDA-GN), which places non-informative priors over these
hyper-parameters and uses Gibbs sampling to learn appropriate values for them.
At the heart of LDA-GN is our proposed 'Gibbs-Newton' algorithm, which is a new
technique for learning the parameters of multivariate Polya distributions. We
report Gibbs-Newton performance results compared with two prominent existing
approaches to the latter task: Minka's fixed-point iteration method and the
Moments method. We then evaluate LDA-GN in two ways: (i) by comparing it with
standard LDA in terms of the ability of the resulting topic models to
generalize to unseen documents; (ii) by comparing it with standard LDA in its
performance on a binary classification task.
Content-based Video Indexing and Retrieval Using Corr-LDA
Authors: Rahul Radhakrishnan Iyer, Sanjeel Parekh, Vikas Mohandoss, Anush Ramsurat, Bhiksha Raj, Rita Singh
Date: 2016-02-27 00:00:00
Tasks: Retrieval

Existing video indexing and retrieval methods on popular web-based multimedia sharing websites are based on user-provided sparse tagging. This paper proposes a very specific way of searching for video clips, based on the content of the video. We present our work on Content-based Video Indexing and Retrieval using the Correspondence-Latent Dirichlet Allocation (corr-LDA) probabilistic framework. This is a model that provides for auto-annotation of videos in a database with textual descriptors, and brings the added benefit of utilizing the semantic relations between the content of the video and text. We use the concept-level matching provided by corr-LDA to build correspondences between text and multimedia, with the objective of retrieving content with increased accuracy. In our experiments, we employ only the audio components of the individual recordings and compare our results with an SVM-based approach.
Superresolution of Noisy Remotely Sensed Images Through Directional Representations
Authors: Wojciech Czaja, James M. Murphy, Daniel Weinberg
Date: 2016-02-27 00:00:00
Tasks: Denoising, Edge Detection, SSIM

We develop an algorithm for single-image superresolution of remotely sensed
data, based on the discrete shearlet transform. The shearlet transform extracts
directional features of signals, and is known to provide near-optimally sparse
representations for a broad class of images. This often leads to superior
performance in edge detection and image representation when compared to
isotropic frames. We justify the use of shearlets mathematically, before
presenting a denoising single-image superresolution algorithm that combines the
shearlet transform with sparse mixing estimators (SME). Our algorithm is
compared with a variety of single-image superresolution methods, including
wavelet SME superresolution. Our numerical results demonstrate competitive
performance in terms of PSNR and SSIM.
Towards Neural Knowledge DNA
Authors: Haoxi Zhang, Cesar Sanin, Edward Szczerbicki
Date: 2016-02-27 00:00:00

In this paper, we propose the Neural Knowledge DNA, a framework that tailors
the ideas underlying the success of neural networks to the scope of knowledge
representation. Knowledge representation is a fundamental field that dedicate
to representing information about the world in a form that computer systems can
utilize to solve complex tasks. The proposed Neural Knowledge DNA is designed
to support discovering, storing, reusing, improving, and sharing knowledge
among machines and organisation. It is constructed in a similar fashion of how
DNA formed: built up by four essential elements. As the DNA produces
phenotypes, the Neural Knowledge DNA carries information and knowledge via its
four essential elements, namely, Networks, Experiences, States, and Actions.
Liberating language research from dogmas of the 20th century
Authors: Ramon Ferrer-i-Cancho, Carlos Gómez-Rodríguez
Date: 2015-09-09 00:00:00

A commentary on the article "Large-scale evidence of dependency length
minimization in 37 languages" by Futrell, Mahowald & Gibson (PNAS 2015 112 (33)
10336-10341).
Multiplier-less Artificial Neurons Exploiting Error Resiliency for Energy-Efficient Neural Computing
Authors: Syed Shakib Sarwar, Swagath Venkataramani, Anand Raghunathan, Kaushik Roy
Date: 2016-02-27 00:00:00

Large-scale artificial neural networks have shown significant promise in
addressing a wide range of classification and recognition applications.
However, their large computational requirements stretch the capabilities of
computing platforms. The fundamental components of these neural networks are
the neurons and its synapses. The core of a digital hardware neuron consists of
multiplier, accumulator and activation function. Multipliers consume most of
the processing energy in the digital neurons, and thereby in the hardware
implementations of artificial neural networks. We propose an approximate
multiplier that utilizes the notion of computation sharing and exploits error
resilience of neural network applications to achieve improved energy
consumption. We also propose Multiplier-less Artificial Neuron (MAN) for even
larger improvement in energy consumption and adapt the training process to
ensure minimal degradation in accuracy. We evaluated the proposed design on 5
recognition applications. The results show, 35% and 60% reduction in energy
consumption, for neuron sizes of 8 bits and 12 bits, respectively, with a
maximum of ~2.83% loss in network accuracy, compared to a conventional neuron
implementation. We also achieve 37% and 62% reduction in area for a neuron size
of 8 bits and 12 bits, respectively, under iso-speed conditions.
Significance Driven Hybrid 8T-6T SRAM for Energy-Efficient Synaptic Storage in Artificial Neural Networks
Authors: Gopalakrishnan Srinivasan, Parami Wijesinghe, Syed Shakib Sarwar, Akhilesh Jaiswal, Kaushik Roy
Date: 2016-02-27 00:00:00
Tasks: General Classification

Multilayered artificial neural networks (ANN) have found widespread utility
in classification and recognition applications. The scale and complexity of
such networks together with the inadequacies of general purpose computing
platforms have led to a significant interest in the development of efficient
hardware implementations. In this work, we focus on designing energy efficient
on-chip storage for the synaptic weights. In order to minimize the power
consumption of typical digital CMOS implementations of such large-scale
networks, the digital neurons could be operated reliably at scaled voltages by
reducing the clock frequency. On the contrary, the on-chip synaptic storage
designed using a conventional 6T SRAM is susceptible to bitcell failures at
reduced voltages. However, the intrinsic error resiliency of NNs to small
synaptic weight perturbations enables us to scale the operating voltage of the
6TSRAM. Our analysis on a widely used digit recognition dataset indicates that
the voltage can be scaled by 200mV from the nominal operating voltage (950mV)
for practically no loss (less than 0.5%) in accuracy (22nm predictive
technology). Scaling beyond that causes substantial performance degradation
owing to increased probability of failures in the MSBs of the synaptic weights.
We, therefore propose a significance driven hybrid 8T-6T SRAM, wherein the
sensitive MSBs are stored in 8T bitcells that are robust at scaled voltages due
to decoupled read and write paths. In an effort to further minimize the area
penalty, we present a synaptic-sensitivity driven hybrid memory architecture
consisting of multiple 8T-6T SRAM banks. Our circuit to system-level simulation
framework shows that the proposed synaptic-sensitivity driven architecture
provides a 30.91% reduction in the memory access power with a 10.41% area
overhead, for less than 1% loss in the classification accuracy.
Prediction of future hospital admissions - what is the tradeoff between specificity and accuracy?
Authors: Ieva Vasiljeva, Ognjen Arandjelovic
Date: 2016-02-27 00:00:00
Tasks: Prediction, Specificity

Large amounts of electronic medical records collected by hospitals across the
developed world offer unprecedented possibilities for knowledge discovery using
computer based data mining and machine learning. Notwithstanding significant
research efforts, the use of this data in the prediction of disease development
has largely been disappointing. In this paper we examine in detail a recently
proposed method which has in preliminary experiments demonstrated highly
promising results on real-world data. We scrutinize the authors' claims that
the proposed model is scalable and investigate whether the tradeoff between
prediction specificity (i.e. the ability of the model to predict a wide number
of different ailments) and accuracy (i.e. the ability of the model to make the
correct prediction) is practically viable. Our experiments conducted on a data
corpus of nearly 3,000,000 admissions support the authors' expectations and
demonstrate that the high prediction accuracy is maintained well even when the
number of admission types explicitly included in the model is increased to
account for 98% of all admissions in the corpus. Thus several promising
directions for future work are highlighted.
Deep multi-scale video prediction beyond mean square error
Authors: Michael Mathieu, Camille Couprie, Yann Lecun
Date: 2015-11-17 00:00:00
Tasks: Optical Flow Estimation, Prediction, Video Prediction

Learning to predict future images from a video sequence involves the
construction of an internal representation that models the image evolution
accurately, and therefore, to some degree, its content and dynamics. This is
why pixel-space video prediction may be viewed as a promising avenue for
unsupervised feature learning. In addition, while optical flow has been a very
studied problem in computer vision for a long time, future frame prediction is
rarely approached. Still, many vision applications could benefit from the
knowledge of the next frames of videos, that does not require the complexity of
tracking every pixel trajectories. In this work, we train a convolutional
network to generate future frames given an input sequence. To deal with the
inherently blurry predictions obtained from the standard Mean Squared Error
(MSE) loss function, we propose three different and complementary feature
learning strategies: a multi-scale architecture, an adversarial training
method, and an image gradient difference loss function. We compare our
predictions to different published results based on recurrent neural networks
on the UCF101 dataset
Patch-Ordering as a Regularization for Inverse Problems in Image Processing
Authors: Gregory Vaksman, Michael Zibulevsky, Michael Elad
Date: 2016-02-26 00:00:00
Tasks: Deblurring, Denoising, Image Deblurring, Image Denoising, Image Restoration, Image Super-Resolution, Super-Resolution

Recent work in image processing suggests that operating on (overlapping)
patches in an image may lead to state-of-the-art results. This has been
demonstrated for a variety of problems including denoising, inpainting,
deblurring, and super-resolution. The work reported in [1,2] takes an extra
step forward by showing that ordering these patches to form an approximate
shortest path can be leveraged for better processing. The core idea is to apply
a simple filter on the resulting 1D smoothed signal obtained after the
patch-permutation. This idea has been also explored in combination with a
wavelet pyramid, leading eventually to a sophisticated and highly effective
regularizer for inverse problems in imaging. In this work we further study the
patch-permutation concept, and harness it to propose a new simple yet effective
regularization for image restoration problems. Our approach builds on the
classic Maximum A'posteriori probability (MAP), with a penalty function
consisting of a regular log-likelihood term and a novel permutation-based
regularization term. Using a plain 1D Laplacian, the proposed regularization
forces robust smoothness (L1) on the permuted pixels. Since the permutation
originates from patch-ordering, we propose to accumulate the smoothness terms
over all the patches' pixels. Furthermore, we take into account the found
distances between adjacent patches in the ordering, by weighting the Laplacian
outcome. We demonstrate the proposed scheme on a diverse set of problems: (i)
severe Poisson image denoising, (ii) Gaussian image denoising, (iii) image
deblurring, and (iv) single image super-resolution. In all these cases, we use
recent methods that handle these problems as initialization to our scheme. This
is followed by an L-BFGS optimization of the above-described penalty function,
leading to state-of-the-art results, and especially so for highly ill-posed
cases.
Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control
Authors: Prashanth L. A., Cheng Jie, Michael Fu, Steve Marcus, Csaba Szepesvári
Date: 2015-06-08 00:00:00
Tasks: Prediction, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Traffic Signal Control

Cumulative prospect theory (CPT) is known to model human decisions well, with
substantial empirical evidence supporting this claim. CPT works by distorting
probabilities and is more general than the classic expected utility and
coherent risk measures. We bring this idea to a risk-sensitive reinforcement
learning (RL) setting and design algorithms for both estimation and control.
The RL setting presents two particular challenges when CPT is applied:
estimating the CPT objective requires estimations of the entire distribution of
the value function and finding a randomized optimal policy. The estimation
scheme that we propose uses the empirical distribution to estimate the
CPT-value of a random variable. We then use this scheme in the inner loop of a
CPT-value optimization procedure that is based on the well-known simulation
optimization idea of simultaneous perturbation stochastic approximation (SPSA).
We provide theoretical convergence guarantees for all the proposed algorithms
and also illustrate the usefulness of CPT-based criteria in a traffic signal
control application.
A Roadmap towards Machine Intelligence
Authors: Tomas Mikolov, Armand Joulin, Marco Baroni
Date: 2015-11-25 00:00:00

The development of intelligent machines is one of the biggest unsolved
challenges in computer science. In this paper, we propose some fundamental
properties these machines should have, focusing in particular on communication
and learning. We discuss a simple environment that could be used to
incrementally teach a machine the basics of natural-language-based
communication, as a prerequisite to more complex interaction with human users.
We also present some conjectures on the sort of algorithms the machine should
support in order to profitably learn from the environment.
Variance Reduced Stochastic Gradient Descent with Neighbors
Authors: Thomas Hofmann, Aurelien Lucchi, Simon Lacoste-Julien, Brian McWilliams
Date: 2015-06-11 00:00:00
Proceeding: NeurIPS 2015 12
Tasks: Memorization

Stochastic Gradient Descent (SGD) is a workhorse in machine learning, yet its
slow convergence can be a computational bottleneck. Variance reduction
techniques such as SAG, SVRG and SAGA have been proposed to overcome this
weakness, achieving linear convergence. However, these methods are either based
on computations of full gradients at pivot points, or on keeping per data point
corrections in memory. Therefore speed-ups relative to SGD may need a minimal
number of epochs in order to materialize. This paper investigates algorithms
that can exploit neighborhood structure in the training data to share and
re-use information about past stochastic gradients across data points, which
offers advantages in the transient optimization phase. As a side-product we
provide a unified convergence analysis for a family of variance reduction
algorithms, which we call memorization algorithms. We provide experimental
results supporting our theory.