pwc-archive/papers-with-abstracts huggingface dataset:
Can Transfer Entropy Infer Information Flow in Neuronal Circuits for Cognitive Processing?
Authors: Ali Tehrani-Saleh, Christoph Adami
Date: 2019-01-22 00:00:00
Tasks: Motion Detection, Time Series Analysis

To infer information flow in any network of agents, it is important first and foremost to establish causal temporal relations between the nodes. Practical and automated methods that can infer causality are difficult to find, and the subject of ongoing research. While Shannon information only detects correlation, there are several information-theoretic notions of "directed information" that have successfully detected causality in some systems, in particular in the neuroscience community. However, recent work has shown that some directed information measures can sometimes inadequately estimate the extent of causal relations, or even fail to identify existing cause-effect relations between components of systems, especially if neurons contribute in a cryptographic manner to influence the effector neuron. Here, we test how often cryptographic logic emerges in an evolutionary process that generates artificial neural circuits for two fundamental cognitive tasks: motion detection and sound localization. We also test whether activity time-series recorded from behaving digital brains can infer information flow using the transfer entropy concept, when compared to a ground-truth model of causal influence constructed from connectivity and circuit logic. Our results suggest that transfer entropy will sometimes fail to infer causality when it exists, and sometimes suggest a causal connection when there is none. However, the extent of incorrect inference strongly depends on the cognitive task considered. These results emphasize the importance of understanding the fundamental logic processes that contribute to information flow in cognitive processing, and quantifying their relevance in any given nervous system.
ICLabel: An automated electroencephalographic independent component classifier, dataset, and website
Authors: Luca Pion-Tonachini, Ken Kreutz-Delgado, Scott Makeig
Date: 2019-01-22 00:00:00
Tasks: Computational Efficiency, EEG, Electroencephalogram (EEG)

The electroencephalogram (EEG) provides a non-invasive, minimally
restrictive, and relatively low cost measure of mesoscale brain dynamics with
high temporal resolution. Although signals recorded in parallel by multiple,
near-adjacent EEG scalp electrode channels are highly-correlated and combine
signals from many different sources, biological and non-biological, independent
component analysis (ICA) has been shown to isolate the various source generator
processes underlying those recordings. Independent components (IC) found by ICA
decomposition can be manually inspected, selected, and interpreted, but doing
so requires both time and practice as ICs have no particular order or intrinsic
interpretations and therefore require further study of their properties.
Alternatively, sufficiently-accurate automated IC classifiers can be used to
classify ICs into broad source categories, speeding the analysis of EEG studies
with many subjects and enabling the use of ICA decomposition in near-real-time
applications. While many such classifiers have been proposed recently, this
work presents the ICLabel project comprised of (1) an IC dataset containing
spatiotemporal measures for over 200,000 ICs from more than 6,000 EEG
recordings, (2) a website for collecting crowdsourced IC labels and educating
EEG researchers and practitioners about IC interpretation, and (3) the
automated ICLabel classifier. The classifier improves upon existing methods in
two ways: by improving the accuracy of the computed label estimates and by
enhancing its computational efficiency. The ICLabel classifier outperforms or
performs comparably to the previous best publicly available method for all
measured IC categories while computing those labels ten times faster than that
classifier as shown in a rigorous comparison against all other publicly
available EEG IC classifiers.
A New CGAN Technique for Constrained Topology Design Optimization
Authors: M. -H. Herman Shen, Liang Chen
Date: 2019-01-22 00:00:00

This paper presents a new conditional GAN (named convex relaxing CGAN or
crCGAN) to replicate the conventional constrained topology optimization
algorithms in an extremely effective and efficient process. The proposed crCGAN
consists of a generator and a discriminator, both of which are deep
convolutional neural networks (CNN) and the topology design constraint can be
conditionally set to both the generator and discriminator. In order to improve
the training efficiency and accuracy due to the dependency between the training
images and the condition, a variety of crCGAN formulation are introduced to
relax the non-convex design space. These new formulations were evaluated and
validated via a series of comprehensive experiments. Moreover, a minibatch
discrimination technique was introduced in the crCGAN training process to
stabilize the convergence and avoid the mode collapse problems. Additional
verifications were conducted using the state-of-the-art MNIST digits and
CIFAR-10 images conditioned by class labels. The experimental evaluations
clearly reveal that the new objective formulation with the minibatch
discrimination training provides not only the accuracy but also the consistency
of the designs.
Predicting Tactical Solutions to Operational Planning Problems under Imperfect Information
Authors: Eric Larsen, Sébastien Lachapelle, Yoshua Bengio, Emma Frejinger, Simon Lacoste-Julien, Andrea Lodi
Date: 2019-01-22 00:00:00
Tasks: BIG-bench Machine Learning, Management

This paper offers a methodological contribution at the intersection of machine learning and operations research. Namely, we propose a methodology to quickly predict tactical solutions to a given operational problem. In this context, the tactical solution is less detailed than the operational one but it has to be computed in very short time and under imperfect information. The problem is of importance in various applications where tactical and operational planning problems are interrelated and information about the operational problem is revealed over time. This is for instance the case in certain capacity planning and demand management systems. We formulate the problem as a two-stage optimal prediction stochastic program whose solution we predict with a supervised machine learning algorithm. The training data set consists of a large number of deterministic (second stage) problems generated by controlled probabilistic sampling. The labels are computed based on solutions to the deterministic problems (solved independently and offline) employing appropriate aggregation and subselection methods to address uncertainty. Results on our motivating application in load planning for rail transportation show that deep learning algorithms produce highly accurate predictions in very short computing time (milliseconds or less). The prediction accuracy is comparable to solutions computed by sample average approximation of the stochastic program.
Modeling Human Motion with Quaternion-based Neural Networks
Authors: Dario Pavllo, Christoph Feichtenhofer, Michael Auli, David Grangier
Date: 2019-01-21 00:00:00

Previous work on predicting or generating 3D human pose sequences regresses either joint rotations or joint positions. The former strategy is prone to error accumulation along the kinematic chain, as well as discontinuities when using Euler angles or exponential maps as parameterizations. The latter requires re-projection onto skeleton constraints to avoid bone stretching and invalid configurations. This work addresses both limitations. QuaterNet represents rotations with quaternions and our loss function performs forward kinematics on a skeleton to penalize absolute position errors instead of angle errors. We investigate both recurrent and convolutional architectures and evaluate on short-term prediction and long-term generation. For the latter, our approach is qualitatively judged as realistic as recent neural strategies from the graphics literature. Our experiments compare quaternions to Euler angles as well as exponential maps and show that only a very short context is required to make reliable future predictions. Finally, we show that the standard evaluation protocol for Human3.6M produces high variance results and we propose a simple solution.
Unsupervised Learning of Neural Networks to Explain Neural Networks (extended abstract)
Authors: Quanshi Zhang, Yu Yang, Ying Nian Wu
Date: 2019-01-21 00:00:00
Tasks: Knowledge Distillation, Object

This paper presents an unsupervised method to learn a neural network, namely
an explainer, to interpret a pre-trained convolutional neural network (CNN),
i.e., the explainer uses interpretable visual concepts to explain features in
middle conv-layers of a CNN. Given feature maps of a conv-layer of the CNN, the
explainer performs like an auto-encoder, which decomposes the feature maps into
object-part features. The object-part features are learned to reconstruct CNN
features without much loss of information. We can consider the disentangled
representations of object parts a paraphrase of CNN features, which help people
understand the knowledge encoded by the CNN. More crucially, we learn the
explainer via knowledge distillation without using any annotations of object
parts or textures for supervision. In experiments, our method was widely used
to interpret features of different benchmark CNNs, and explainers significantly
boosted the feature interpretability without hurting the discrimination power
of the CNNs.
Neural Decoder for Topological Codes using Pseudo-Inverse of Parity Check Matrix
Authors: Chaitanya Chinni, Abhishek Kulkarni, Dheeraj M. Pai, Kaushik Mitra, Pradeep Kiran Sarvepalli
Date: 2019-01-21 00:00:00
Tasks: Decoder

Recent developments in the field of deep learning have motivated many
researchers to apply these methods to problems in quantum information. Torlai
and Melko first proposed a decoder for surface codes based on neural networks.
Since then, many other researchers have applied neural networks to study a
variety of problems in the context of decoding. An important development in
this regard was due to Varsamopoulos et al. who proposed a two-step decoder
using neural networks. Subsequent work of Maskara et al. used the same concept
for decoding for various noise models. We propose a similar two-step neural
decoder using inverse parity-check matrix for topological color codes. We show
that it outperforms the state-of-the-art performance of non-neural decoders for
independent Pauli errors noise model on a 2D hexagonal color code. Our final
decoder is independent of the noise model and achieves a threshold of $10 \%$.
Our result is comparable to the recent work on neural decoder for quantum error
correction by Maskara et al.. It appears that our decoder has significant
advantages with respect to training cost and complexity of the network for
higher lengths when compared to that of Maskara et al.. Our proposed method can
also be extended to arbitrary dimension and other stabilizer codes.
A Machine Learning Benchmark for Facies Classification
Authors: Yazeed Alaudah, Patrycja Michalowicz, Motaz Alfarraj, Ghassan AlRegib
Date: 2019-01-12 00:00:00
Tasks: BIG-bench Machine Learning, Classification, Facies Classification, General Classification, Seismic Interpretation

The recent interest in using deep learning for seismic interpretation tasks,
such as facies classification, has been facing a significant obstacle, namely
the absence of large publicly available annotated datasets for training and
testing models. As a result, researchers have often resorted to annotating
their own training and testing data. However, different researchers may
annotate different classes, or use different train and test splits. In
addition, it is common for papers that apply machine learning for facies
classification to not contain quantitative results, and rather rely solely on
visual inspection of the results. All of these practices have lead to
subjective results and have greatly hindered the ability to compare different
machine learning models against each other and understand the advantages and
disadvantages of each approach.
  To address these issues, we open-source a fully-annotated 3D geological model
of the Netherlands F3 Block. This model is based on the study of the 3D seismic
data in addition to 26 well logs, and is grounded on the careful study of the
geology of the region. Furthermore, we propose two baseline models for facies
classification based on a deconvolution network architecture and make their
codes publicly available. Finally, we propose a scheme for evaluating different
models on this dataset, and we share the results of our baseline models. In
addition to making the dataset and the code publicly available, this work helps
advance research in this area by creating an objective benchmark for comparing
the results of different machine learning approaches for facies classification.
Semantic Relation Classification via Bidirectional LSTM Networks with Entity-aware Attention using Latent Entity Typing
Authors: Joohong Lee, Sangwoo Seo, Yong Suk Choi
Date: 2019-01-23 00:00:00
Tasks: Entity Typing, General Classification, Named Entity Recognition (NER), POS, Relation, Relation Classification, Relation Extraction

Classifying semantic relations between entity pairs in sentences is an
important task in Natural Language Processing (NLP). Most previous models for
relation classification rely on the high-level lexical and syntactic features
obtained by NLP tools such as WordNet, dependency parser, part-of-speech (POS)
tagger, and named entity recognizers (NER). In addition, state-of-the-art
neural models based on attention mechanisms do not fully utilize information of
entity that may be the most crucial features for relation classification. To
address these issues, we propose a novel end-to-end recurrent neural model
which incorporates an entity-aware attention mechanism with a latent entity
typing (LET) method. Our model not only utilizes entities and their latent
types as features effectively but also is more interpretable by visualizing
attention mechanisms applied to our model and results of LET. Experimental
results on the SemEval-2010 Task 8, one of the most popular relation
classification task, demonstrate that our model outperforms existing
state-of-the-art models without any high-level features.
Multigrid Backprojection Super-Resolution and Deep Filter Visualization
Authors: Pablo Navarrete Michelini, Hanwen Liu, Dan Zhu
Date: 2018-09-25 00:00:00
Tasks: Super-Resolution

We introduce a novel deep-learning architecture for image upscaling by large
factors (e.g. 4x, 8x) based on examples of pristine high-resolution images. Our
target is to reconstruct high-resolution images from their downscale versions.
The proposed system performs a multi-level progressive upscaling, starting from
small factors (2x) and updating for higher factors (4x and 8x). The system is
recursive as it repeats the same procedure at each level. It is also residual
since we use the network to update the outputs of a classic upscaler. The
network residuals are improved by Iterative Back-Projections (IBP) computed in
the features of a convolutional network. To work in multiple levels we extend
the standard back-projection algorithm using a recursion analogous to
Multi-Grid algorithms commonly used as solvers of large systems of linear
equations. We finally show how the network can be interpreted as a standard
upsampling-and-filter upscaler with a space-variant filter that adapts to the
geometry. This approach allows us to visualize how the network learns to
upscale. Finally, our system reaches state of the art quality for models with
relatively few number of parameters.
A Knowledge-Grounded Multimodal Search-Based Conversational Agent
Authors: Shubham Agarwal, Ondrej Dusek, Ioannis Konstas, Verena Rieser
Date: 2018-10-20 00:00:00
Proceeding: WS 2018 10
Tasks: Decoder, Question Answering, Response Generation

Multimodal search-based dialogue is a challenging new task: It extends
visually grounded question answering systems into multi-turn conversations with
access to an external database. We address this new challenge by learning a
neural response generation system from the recently released Multimodal
Dialogue (MMD) dataset (Saha et al., 2017). We introduce a knowledge-grounded
multimodal conversational model where an encoded knowledge base (KB)
representation is appended to the decoder input. Our model substantially
outperforms strong baselines in terms of text-based similarity measures (over 9
BLEU points, 3 of which are solely due to the use of additional information
from the KB.
Improving Context Modelling in Multimodal Dialogue Generation
Authors: Shubham Agarwal, Ondrej Dusek, Ioannis Konstas, Verena Rieser
Date: 2018-10-20 00:00:00
Proceeding: WS 2018 11
Tasks: Decoder, Dialogue Generation, Response Generation

In this work, we investigate the task of textual response generation in a
multimodal task-oriented dialogue system. Our work is based on the recently
released Multimodal Dialogue (MMD) dataset (Saha et al., 2017) in the fashion
domain. We introduce a multimodal extension to the Hierarchical Recurrent
Encoder-Decoder (HRED) model and show that this extension outperforms strong
baselines in terms of text-based similarity metrics. We also showcase the
shortcomings of current vision and language models by performing an error
analysis on our system's output.
Sparse Subspace Clustering: Algorithm, Theory, and Applications
Authors: Ehsan Elhamifar, Rene Vidal
Date: 2012-03-05 00:00:00
Tasks: Clustering, Face Clustering, Image Clustering, Motion Segmentation

In many real-world problems, we are dealing with collections of
high-dimensional data, such as images, videos, text and web documents, DNA
microarray data, and more. Often, high-dimensional data lie close to
low-dimensional structures corresponding to several classes or categories the
data belongs to. In this paper, we propose and study an algorithm, called
Sparse Subspace Clustering (SSC), to cluster data points that lie in a union of
low-dimensional subspaces. The key idea is that, among infinitely many possible
representations of a data point in terms of other points, a sparse
representation corresponds to selecting a few points from the same subspace.
This motivates solving a sparse optimization program whose solution is used in
a spectral clustering framework to infer the clustering of data into subspaces.
Since solving the sparse optimization program is in general NP-hard, we
consider a convex relaxation and show that, under appropriate conditions on the
arrangement of subspaces and the distribution of data, the proposed
minimization program succeeds in recovering the desired sparse representations.
The proposed algorithm can be solved efficiently and can handle data points
near the intersections of subspaces. Another key advantage of the proposed
algorithm with respect to the state of the art is that it can deal with data
nuisances, such as noise, sparse outlying entries, and missing entries,
directly by incorporating the model of the data into the sparse optimization
program. We demonstrate the effectiveness of the proposed algorithm through
experiments on synthetic data as well as the two real-world problems of motion
segmentation and face clustering.
Graph heat mixture model learning
Authors: Hermina Petric Maretic, Mireille El Gheche, Pascal Frossard
Date: 2019-01-24 00:00:00
Tasks: model

Graph inference methods have recently attracted a great interest from the
scientific community, due to the large value they bring in data interpretation
and analysis. However, most of the available state-of-the-art methods focus on
scenarios where all available data can be explained through the same graph, or
groups corresponding to each graph are known a priori. In this paper, we argue
that this is not always realistic and we introduce a generative model for mixed
signals following a heat diffusion process on multiple graphs. We propose an
expectation-maximisation algorithm that can successfully separate signals into
corresponding groups, and infer multiple graphs that govern their behaviour. We
demonstrate the benefits of our method on both synthetic and real data.
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
Authors: Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruosong Wang
Date: 2019-01-24 00:00:00

Recent works have cast some light on the mystery of why deep nets fit any data and generalize despite being very overparametrized. This paper analyzes training and generalization for a simple 2-layer ReLU net with random initialization, and provides the following improvements over recent works: (i) Using a tighter characterization of training speed than recent papers, an explanation for why training a neural net with random labels leads to slower training, as originally observed in [Zhang et al. ICLR'17]. (ii) Generalization bound independent of network size, using a data-dependent complexity measure. Our measure distinguishes clearly between random labels and true labels on MNIST and CIFAR, as shown by experiments. Moreover, recent papers require sample complexity to increase (slowly) with the size, while our sample complexity is completely independent of the network size. (iii) Learnability of a broad class of smooth functions by 2-layer ReLU nets trained via gradient descent. The key idea is to track dynamics of training and generalization via properties of a related kernel.
Memory-free dynamics for the TAP equations of Ising models with arbitrary rotation invariant ensembles of random coupling matrices
Authors: Burak Çakmak, Manfred Opper
Date: 2019-01-24 00:00:00

We propose an iterative algorithm for solving the Thouless-Anderson-Palmer
(TAP) equations of Ising models with arbitrary rotation invariant (random)
coupling matrices. In the thermodynamic limit, we prove by means of the
dynamical functional method that the proposed algorithm converges when the
so-called de Almeida Thouless (AT) criterion is fulfilled. Moreover, we give
exact analytical expressions for the rate of the convergence.
Forecasting Transformative AI: An Expert Survey
Authors: Ross Gruetzemacher, David Paradice, Kang Bok Lee
Date: 2019-01-24 00:00:00
Tasks: Survey

Transformative AI technologies have the potential to reshape critical aspects of society in the near future. However, in order to properly prepare policy initiatives for the arrival of such technologies accurate forecasts and timelines are necessary. A survey was administered to attendees of three AI conferences during the summer of 2018 (ICML, IJCAI and the HLAI conference). The survey included questions for estimating AI capabilities over the next decade, questions for forecasting five scenarios of transformative AI and questions concerning the impact of computational resources in AI research. Respondents indicated a median of 21.5% of human tasks (i.e., all tasks that humans are currently paid to do) can be feasibly automated now, and that this figure would rise to 40% in 5 years and 60% in 10 years. Median forecasts indicated a 50% probability of AI systems being capable of automating 90% of current human tasks in 25 years and 99% of current human tasks in 50 years. The conference of attendance was found to have a statistically significant impact on all forecasts, with attendees of HLAI providing more optimistic timelines with less uncertainty. These findings suggest that AI experts expect major advances in AI technology to continue over the next decade to a degree that will likely have profound transformative impacts on society.