pwc-archive/papers-with-abstracts huggingface dataset:
Principles of design and software development models of ontological-driven computer systems
Authors: A. V. Palagin, N. G. Petrenko, V. Yu. Velychko, K. S. Malakhov, O. V. Karun
Date: 2018-02-13 00:00:00

This paper describes the design principles of methodology of
knowledge-oriented information systems based on ontological approach. Such
systems implement technology subject-oriented extraction of knowledge from the
set of natural language texts and their formal and logical presentation and
application processing
Tighter Variational Bounds are Not Necessarily Better
Authors: Tom Rainforth, Adam R. Kosiorek, Tuan Anh Le, Chris J. Maddison, Maximilian Igl, Frank Wood, Yee Whye Teh
Date: 2018-02-13 00:00:00
Proceeding: ICML 2018 7

We provide theoretical and empirical evidence that using tighter evidence
lower bounds (ELBOs) can be detrimental to the process of learning an inference
network by reducing the signal-to-noise ratio of the gradient estimator. Our
results call into question common implicit assumptions that tighter ELBOs are
better variational objectives for simultaneous model learning and inference
amortization schemes. Based on our insights, we introduce three new algorithms:
the partially importance weighted auto-encoder (PIWAE), the multiply importance
weighted auto-encoder (MIWAE), and the combination importance weighted
auto-encoder (CIWAE), each of which includes the standard importance weighted
auto-encoder (IWAE) as a special case. We show that each can deliver
improvements over IWAE, even when performance is measured by the IWAE target
itself. Furthermore, our results suggest that PIWAE may be able to deliver
simultaneous improvements in the training of both the inference and generative
networks.
Distributed Bayesian Matrix Factorization with Limited Communication
Authors: Xiangju Qin, Paul Blomstedt, Eemeli Leppäaho, Pekka Parviainen, Samuel Kaski
Date: 2017-03-02 00:00:00
Tasks: Missing Values

Bayesian matrix factorization (BMF) is a powerful tool for producing low-rank
representations of matrices and for predicting missing values and providing
confidence intervals. Scaling up the posterior inference for massive-scale
matrices is challenging and requires distributing both data and computation
over many workers, making communication the main computational bottleneck.
Embarrassingly parallel inference would remove the communication needed, by
using completely independent computations on different data subsets, but it
suffers from the inherent unidentifiability of BMF solutions. We introduce a
hierarchical decomposition of the joint posterior distribution, which couples
the subset inferences, allowing for embarrassingly parallel computations in a
sequence of at most three stages. Using an efficient approximate
implementation, we show improvements empirically on both real and simulated
data. Our distributed approach is able to achieve a speed-up of almost an order
of magnitude over the full posterior, with a negligible effect on predictive
accuracy. Our method outperforms state-of-the-art embarrassingly parallel MCMC
methods in accuracy, and achieves results competitive to other available
distributed and parallel implementations of BMF.
Efficient Algorithms for Searching the Minimum Information Partition in Integrated Information Theory
Authors: Jun Kitazono, Ryota Kanai, Masafumi Oizumi
Date: 2017-12-19 00:00:00

The ability to integrate information in the brain is considered to be an
essential property for cognition and consciousness. Integrated Information
Theory (IIT) hypothesizes that the amount of integrated information ($\Phi$) in
the brain is related to the level of consciousness. IIT proposes that to
quantify information integration in a system as a whole, integrated information
should be measured across the partition of the system at which information loss
caused by partitioning is minimized, called the Minimum Information Partition
(MIP). The computational cost for exhaustively searching for the MIP grows
exponentially with system size, making it difficult to apply IIT to real neural
data. It has been previously shown that if a measure of $\Phi$ satisfies a
mathematical property, submodularity, the MIP can be found in a polynomial
order by an optimization algorithm. However, although the first version of
$\Phi$ is submodular, the later versions are not. In this study, we empirically
explore to what extent the algorithm can be applied to the non-submodular
measures of $\Phi$ by evaluating the accuracy of the algorithm in simulated
data and real neural data. We find that the algorithm identifies the MIP in a
nearly perfect manner even for the non-submodular measures. Our results show
that the algorithm allows us to measure $\Phi$ in large systems within a
practical amount of time.
Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk
Authors: Paul Hand, Vladislav Voroninski
Date: 2017-05-22 00:00:00

We examine the theoretical properties of enforcing priors provided by
generative deep neural networks via empirical risk minimization. In particular
we consider two models, one in which the task is to invert a generative neural
network given access to its last layer and another in which the task is to
invert a generative neural network given only compressive linear observations
of its last layer. We establish that in both cases, in suitable regimes of
network layer sizes and a randomness assumption on the network weights, that
the non-convex objective function given by empirical risk minimization does not
have any spurious stationary points. That is, we establish that with high
probability, at any point away from small neighborhoods around two scalar
multiples of the desired solution, there is a descent direction. Hence, there
are no local minima, saddle points, or other stationary points outside these
neighborhoods. These results constitute the first theoretical guarantees which
establish the favorable global geometry of these non-convex optimization
problems, and they bridge the gap between the empirical success of enforcing
deep generative priors and a rigorous understanding of non-linear inverse
problems.
Query-Free Attacks on Industry-Grade Face Recognition Systems under Resource Constraints
Authors: Di Tang, XiaoFeng Wang, Kehuan Zhang
Date: 2018-02-13 00:00:00
Tasks: Face Recognition

To launch black-box attacks against a Deep Neural Network (DNN) based Face
Recognition (FR) system, one needs to build \textit{substitute} models to
simulate the target model, so the adversarial examples discovered from
substitute models could also mislead the target model. Such
\textit{transferability} is achieved in recent studies through querying the
target model to obtain data for training the substitute models. A real-world
target, likes the FR system of law enforcement, however, is less accessible to
the adversary. To attack such a system, a substitute model with similar quality
as the target model is needed to identify their common defects. This is hard
since the adversary often does not have the enough resources to train such a
powerful model (hundreds of millions of images and rooms of GPUs are needed to
train a commercial FR system).
  We found in our research, however, that a resource-constrained adversary
could still effectively approximate the target model's capability to recognize
\textit{specific} individuals, by training \textit{biased} substitute models on
additional images of those victims whose identities the attacker want to cover
or impersonate. This is made possible by a new property we discovered, called
\textit{Nearly Local Linearity} (NLL), which models the observation that an
ideal DNN model produces the image representations (embeddings) whose distances
among themselves truthfully describe the human perception of the differences
among the input images. By simulating this property around the victim's images,
we significantly improve the transferability of black-box impersonation attacks
by nearly 50\%. Particularly, we successfully attacked a commercial system
trained over 20 million images, using 4 million images and 1/5 of the training
time but achieving 62\% transferability in an impersonation attack and 89\% in
a dodging attack.
Legendre Decomposition for Tensors
Authors: Mahito Sugiyama, Hiroyuki Nakahara, Koji Tsuda
Date: 2018-02-13 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Tensor Decomposition

We present a novel nonnegative tensor decomposition method, called Legendre
decomposition, which factorizes an input tensor into a multiplicative
combination of parameters. Thanks to the well-developed theory of information
geometry, the reconstructed tensor is unique and always minimizes the KL
divergence from an input tensor. We empirically show that Legendre
decomposition can more accurately reconstruct tensors than other nonnegative
tensor decomposition methods.
Attention-Based End-to-End Speech Recognition on Voice Search
Authors: Changhao Shan, Junbo Zhang, Yujun Wang, Lei Xie
Date: 2017-07-22 00:00:00
Tasks: Decoder, L2 Regularization, Language Modeling, Language Modelling, Sentence, speech-recognition, Speech Recognition, Speech-to-Text

Recently, there has been a growing interest in end-to-end speech recognition
that directly transcribes speech to text without any predefined alignments. In
this paper, we explore the use of attention-based encoder-decoder model for
Mandarin speech recognition on a voice search task. Previous attempts have
shown that applying attention-based encoder-decoder to Mandarin speech
recognition was quite difficult due to the logographic orthography of Mandarin,
the large vocabulary and the conditional dependency of the attention model. In
this paper, we use character embedding to deal with the large vocabulary.
Several tricks are used for effective model training, including L2
regularization, Gaussian weight noise and frame skipping. We compare two
attention mechanisms and use attention smoothing to cover long context in the
attention model. Taken together, these tricks allow us to finally achieve a
character error rate (CER) of 3.58% and a sentence error rate (SER) of 7.43% on
the MiTV voice search dataset. While together with a trigram language model,
CER and SER reach 2.81% and 5.77%, respectively.
Latent Relational Metric Learning via Memory-based Attention for Collaborative Ranking
Authors: Yi Tay, Anh Tuan Luu, Siu Cheung Hui
Date: 2017-07-17 00:00:00
Tasks: Attribute, Collaborative Ranking, Metric Learning, Recommendation Systems

This paper proposes a new neural architecture for collaborative ranking with
implicit feedback. Our model, LRML (\textit{Latent Relational Metric Learning})
is a novel metric learning approach for recommendation. More specifically,
instead of simple push-pull mechanisms between user and item pairs, we propose
to learn latent relations that describe each user item interaction. This helps
to alleviate the potential geometric inflexibility of existing metric learing
approaches. This enables not only better performance but also a greater extent
of modeling capability, allowing our model to scale to a larger number of
interactions. In order to do so, we employ a augmented memory module and learn
to attend over these memory blocks to construct latent relations. The
memory-based attention module is controlled by the user-item interaction,
making the learned relation vector specific to each user-item pair. Hence, this
can be interpreted as learning an exclusive and optimal relational translation
for each user-item interaction. The proposed architecture demonstrates the
state-of-the-art performance across multiple recommendation benchmarks. LRML
outperforms other metric learning models by $6\%-7.5\%$ in terms of Hits@10 and
nDCG@10 on large datasets such as Netflix and MovieLens20M. Moreover,
qualitative studies also demonstrate evidence that our proposed model is able
to infer and encode explicit sentiment, temporal and attribute information
despite being only trained on implicit feedback. As such, this ascertains the
ability of LRML to uncover hidden relational structure within implicit
datasets.
Graph-Based Ascent Algorithms for Function Maximization
Authors: Muni Sreenivas Pydi, Varun Jog, Po-Ling Loh
Date: 2018-02-13 00:00:00

We study the problem of finding the maximum of a function defined on the
nodes of a connected graph. The goal is to identify a node where the function
obtains its maximum. We focus on local iterative algorithms, which traverse the
nodes of the graph along a path, and the next iterate is chosen from the
neighbors of the current iterate with probability distribution determined by
the function values at the current iterate and its neighbors. We study two
algorithms corresponding to a Metropolis-Hastings random walk with different
transition kernels: (i) The first algorithm is an exponentially weighted random
walk governed by a parameter $\gamma$. (ii) The second algorithm is defined
with respect to the graph Laplacian and a smoothness parameter $k$. We derive
convergence rates for the two algorithms in terms of total variation distance
and hitting times. We also provide simulations showing the relative convergence
rates of our algorithms in comparison to an unbiased random walk, as a function
of the smoothness of the graph function. Our algorithms may be categorized as a
new class of "descent-based" methods for function maximization on the nodes of
a graph.
Deep Neural Networks Learn Non-Smooth Functions Effectively
Authors: Masaaki Imaizumi, Kenji Fukumizu
Date: 2018-02-13 00:00:00

We theoretically discuss why deep neural networks (DNNs) performs better than
other models in some cases by investigating statistical properties of DNNs for
non-smooth functions. While DNNs have empirically shown higher performance than
other standard methods, understanding its mechanism is still a challenging
problem. From an aspect of the statistical theory, it is known many standard
methods attain the optimal rate of generalization errors for smooth functions
in large sample asymptotics, and thus it has not been straightforward to find
theoretical advantages of DNNs. This paper fills this gap by considering
learning of a certain class of non-smooth functions, which was not covered by
the previous theory. We derive the generalization error of estimators by DNNs
with a ReLU activation, and show that convergence rates of the generalization
by DNNs are almost optimal to estimate the non-smooth functions, while some of
the popular models do not attain the optimal rate. In addition, our theoretical
result provides guidelines for selecting an appropriate number of layers and
edges of DNNs. We provide numerical experiments to support the theoretical
results.
Information Scaling Law of Deep Neural Networks
Authors: Xiao-Yang Liu
Date: 2018-02-13 00:00:00

With the rapid development of Deep Neural Networks (DNNs), various network
models that show strong computing power and impressive expressive power are
proposed. However, there is no comprehensive informational interpretation of
DNNs from the perspective of information theory. Due to the nonlinear function
and the uncertain number of layers and neural units used in the DNNs, the
network structure shows nonlinearity and complexity. With the typical DNNs
named Convolutional Arithmetic Circuits (ConvACs), the complex DNNs can be
converted into mathematical formula. Thus, we can use rigorous mathematical
theory especially the information theory to analyse the complicated DNNs. In
this paper, we propose a novel information scaling law scheme that can
interpret the network's inner organization by information theory. First, we
show the informational interpretation of the activation function. Secondly, we
prove that the information entropy increases when the information is
transmitted through the ConvACs. Finally, we propose the information scaling
law of ConvACs through making a reasonable assumption.
An Optimized Architecture for Unpaired Image-to-Image Translation
Authors: Mohan Nikam
Date: 2018-02-13 00:00:00
Tasks: Image-to-Image Translation, Translation

Unpaired Image-to-Image translation aims to convert the image from one domain
(input domain A) to another domain (target domain B), without providing paired
examples for the training. The state-of-the-art, Cycle-GAN demonstrated the
power of Generative Adversarial Networks with Cycle-Consistency Loss. While its
results are promising, there is scope for optimization in the training process.
This paper introduces a new neural network architecture, which only learns the
translation from domain A to B and eliminates the need for reverse mapping (B
to A), by introducing a new Deviation-loss term. Furthermore, few other
improvements to the Cycle-GAN are found and utilized in this new architecture,
contributing to significantly lesser training duration.
Brain Tumor Segmentation Based on Refined Fully Convolutional Neural Networks with A Hierarchical Dice Loss
Authors: Jiachi Zhang, Xiaolei Shen, Tianqi Zhuo, Hong Zhou
Date: 2017-12-25 00:00:00
Tasks: Binary Classification, Brain Tumor Segmentation, General Classification, Instance Segmentation, object-detection, Object Detection, Segmentation, Semantic Segmentation, Tumor Segmentation

As a basic task in computer vision, semantic segmentation can provide
fundamental information for object detection and instance segmentation to help
the artificial intelligence better understand real world. Since the proposal of
fully convolutional neural network (FCNN), it has been widely used in semantic
segmentation because of its high accuracy of pixel-wise classification as well
as high precision of localization. In this paper, we apply several famous FCNN
to brain tumor segmentation, making comparisons and adjusting network
architectures to achieve better performance measured by metrics such as
precision, recall, mean of intersection of union (mIoU) and dice score
coefficient (DSC). The adjustments to the classic FCNN include adding more
connections between convolutional layers, enlarging decoders after up sample
layers and changing the way shallower layers' information is reused. Besides
the structure modification, we also propose a new classifier with a
hierarchical dice loss. Inspired by the containing relationship between
classes, the loss function converts multiple classification to multiple binary
classification in order to counteract the negative effect caused by imbalance
data set. Massive experiments have been done on the training set and testing
set in order to assess our refined fully convolutional neural networks and new
types of loss function. Competitive figures prove they are more effective than
their predecessors.
Predicting Adversarial Examples with High Confidence
Authors: Angus Galloway, Graham W. Taylor, Medhat Moussa
Date: 2018-02-13 00:00:00
Tasks: Data Augmentation, Diversity, Vocal Bursts Intensity Prediction

It has been suggested that adversarial examples cause deep learning models to
make incorrect predictions with high confidence. In this work, we take the
opposite stance: an overly confident model is more likely to be vulnerable to
adversarial examples. This work is one of the most proactive approaches taken
to date, as we link robustness with non-calibrated model confidence on noisy
images, providing a data-augmentation-free path forward. The adversarial
examples phenomenon is most easily explained by the trend of increasing
non-regularized model capacity, while the diversity and number of samples in
common datasets has remained flat. Test accuracy has incorrectly been
associated with true generalization performance, ignoring that training and
test splits are often extremely similar in terms of the overall representation
space. The transferability property of adversarial examples was previously used
as evidence against overfitting arguments, a perceived random effect, but
overfitting is not always random.
Threshold Auto-Tuning Metric Learning
Authors: Yuya Onuma, Rachelle Rivero, Tsuyoshi Kato
Date: 2018-01-07 00:00:00
Tasks: Metric Learning

It has been reported repeatedly that discriminative learning of distance
metric boosts the pattern recognition performance. A weak point of ITML-based
methods is that the distance threshold for similarity/dissimilarity constraints
must be determined manually and it is sensitive to generalization performance,
although the ITML-based methods enjoy an advantage that the Bregman projection
framework can be applied for optimization of distance metric. In this paper, we
present a new formulation of metric learning algorithm in which the distance
threshold is optimized together. Since the optimization is still in the Bregman
projection framework, the Dykstra algorithm can be applied for optimization. A
nonlinear equation has to be solved to project the solution onto a half-space
in each iteration. Na\"{i}ve method takes $O(LMn^{3})$ computational time to
solve the nonlinear equation. In this study, an efficient technique that can
solve the nonlinear equation in $O(Mn^{3})$ has been discovered. We have proved
that the root exists and is unique. We empirically show that the accuracy of
pattern recognition for the proposed metric learning algorithm is comparable to
the existing metric learning methods, yet the distance threshold is
automatically tuned for the proposed metric learning algorithm.
Blockchain and Artificial Intelligence
Authors: Tshilidzi Marwala, Bo Xing
Date: 2018-02-13 00:00:00

It is undeniable that artificial intelligence (AI) and blockchain concepts
are spreading at a phenomenal rate. Both technologies have distinct degree of
technological complexity and multi-dimensional business implications. However,
a common misunderstanding about blockchain concept, in particular, is that
blockchain is decentralized and is not controlled by anyone. But the underlying
development of a blockchain system is still attributed to a cluster of core
developers. Take smart contract as an example, it is essentially a collection
of codes (or functions) and data (or states) that are programmed and deployed
on a blockchain (say, Ethereum) by different human programmers. It is thus,
unfortunately, less likely to be free of loopholes and flaws. In this article,
through a brief overview about how artificial intelligence could be used to
deliver bug-free smart contract so as to achieve the goal of blockchain 2.0, we
to emphasize that the blockchain implementation can be assisted or enhanced via
various AI techniques. The alliance of AI and blockchain is expected to create
numerous possibilities.
On Characterizing the Capacity of Neural Networks using Algebraic Topology
Authors: William H. Guss, Ruslan Salakhutdinov
Date: 2018-02-13 00:00:00
Proceeding: ICLR 2018 1
Tasks: Inductive Bias

The learnability of different neural architectures can be characterized
directly by computable measures of data complexity. In this paper, we reframe
the problem of architecture selection as understanding how data determines the
most expressive and generalizable architectures suited to that data, beyond
inductive bias. After suggesting algebraic topology as a measure for data
complexity, we show that the power of a network to express the topological
complexity of a dataset in its decision region is a strictly limiting factor in
its ability to generalize. We then provide the first empirical characterization
of the topological capacity of neural networks. Our empirical analysis shows
that at every level of dataset complexity, neural networks exhibit topological
phase transitions. This observation allowed us to connect existing theory to
empirically driven conjectures on the choice of architectures for
fully-connected neural networks.