pwc-archive/papers-with-abstracts huggingface dataset:
Local Aggregation for Unsupervised Learning of Visual Embeddings
Authors: Chengxu Zhuang, Alex Lin Zhai, Daniel Yamins
Date: 2019-03-29 00:00:00
Proceeding: ICCV 2019 10
Tasks: Clustering, Contrastive Learning, object-detection, Object Detection, Object Recognition, Scene Recognition, Self-Supervised Image Classification, Transfer Learning

Unsupervised approaches to learning in neural networks are of substantial
interest for furthering artificial intelligence, both because they would enable
the training of networks without the need for large numbers of expensive
annotations, and because they would be better models of the kind of
general-purpose learning deployed by humans. However, unsupervised networks
have long lagged behind the performance of their supervised counterparts,
especially in the domain of large-scale visual recognition. Recent developments
in training deep convolutional embeddings to maximize non-parametric instance
separation and clustering objectives have shown promise in closing this gap.
Here, we describe a method that trains an embedding function to maximize a
metric of local aggregation, causing similar data instances to move together in
the embedding space, while allowing dissimilar instances to separate. This
aggregation metric is dynamic, allowing soft clusters of different scales to
emerge. We evaluate our procedure on several large-scale visual recognition
datasets, achieving state-of-the-art unsupervised transfer learning performance
on object recognition in ImageNet, scene recognition in Places 205, and object
detection in PASCAL VOC.
A General FOFE-net Framework for Simple and Effective Question Answering over Knowledge Bases
Authors: Dekun Wu, Nana Nosirova, Hui Jiang, MingBin Xu
Date: 2019-03-29 00:00:00
Tasks: Question Answering, Relation

Question answering over knowledge base (KB-QA) has recently become a popular
research topic in NLP. One popular way to solve the KB-QA problem is to make
use of a pipeline of several NLP modules, including entity discovery and
linking (EDL) and relation detection. Recent success on KB-QA task usually
involves complex network structures with sophisticated heuristics. Inspired by
a previous work that builds a strong KB-QA baseline, we propose a simple but
general neural model composed of fixed-size ordinally forgetting encoding
(FOFE) and deep neural networks, called FOFE-net to solve KB-QA problem at
different stages. For evaluation, we use two popular KB-QA datasets,
SimpleQuestions and WebQSP, and a newly created dataset, FreebaseQA. The
experimental results show that FOFE-net performs well on KB-QA subtasks, entity
discovery and linking (EDL) and relation detection, and in turn pushing overall
KB-QA system to achieve strong results on all datasets.
CUTIE: Learning to Understand Documents with Convolutional Universal Text Information Extractor
Authors: Xiaohui Zhao, Endi Niu, Zhuo Wu, Xiaoguang Wang
Date: 2019-03-29 00:00:00
Tasks: Named Entity Recognition, Named Entity Recognition (NER)

Extracting key information from documents, such as receipts or invoices, and preserving the interested texts to structured data is crucial in the document-intensive streamline processes of office automation in areas that includes but not limited to accounting, financial, and taxation areas. To avoid designing expert rules for each specific type of document, some published works attempt to tackle the problem by learning a model to explore the semantic context in text sequences based on the Named Entity Recognition (NER) method in the NLP field. In this paper, we propose to harness the effective information from both semantic meaning and spatial distribution of texts in documents. Specifically, our proposed model, Convolutional Universal Text Information Extractor (CUTIE), applies convolutional neural networks on gridded texts where texts are embedded as features with semantical connotations. We further explore the effect of employing different structures of convolutional neural network and propose a fast and portable structure. We demonstrate the effectiveness of the proposed method on a dataset with up to $4,484$ labelled receipts, without any pre-training or post-processing, achieving state of the art performance that is much better than the NER based methods in terms of either speed and accuracy. Experimental results also demonstrate that the proposed CUTIE model being able to achieve good performance with a much smaller amount of training data.
Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
Authors: Laura von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, Annika Pick, Rajkumar Ramamurthy, Michal Walczak, Jochen Garcke, Christian Bauckhage, Jannis Schuecker
Date: 2019-03-29 00:00:00
Tasks: BIG-bench Machine Learning, General Classification

Despite its great success, machine learning can have its limits when dealing with insufficient training data. A potential solution is the additional integration of prior knowledge into the training process which leads to the notion of informed machine learning. In this paper, we present a structured overview of various approaches in this field. We provide a definition and propose a concept for informed machine learning which illustrates its building blocks and distinguishes it from conventional machine learning. We introduce a taxonomy that serves as a classification framework for informed machine learning approaches. It considers the source of knowledge, its representation, and its integration into the machine learning pipeline. Based on this taxonomy, we survey related research and describe how different knowledge representations such as algebraic equations, logic rules, or simulation results can be used in learning systems. This evaluation of numerous papers on the basis of our taxonomy uncovers key methods in the field of informed machine learning.
Proceedings 7th International Workshop on Theorem proving components for Educational software
Authors: Pedro Quaresma, Walther Neuper
Date: 2019-03-29 00:00:00
Tasks: Automated Theorem Proving

The 7th International Workshop on Theorem proving components for Educational
software (ThEdu'18) was held in Oxford, United Kingdom, on 18 July 2018. It was
associated to the conference, Federated Logic Conference 2018 (FLoC2018).
  The major aim of the ThEdu workshop series was to link developers interested
in adapting Computer Theorem Proving (TP) to the needs of education and to
inform mathematicians and mathematics educators about TP's potential for
educational software. Topics of interest include: methods of automated
deduction applied to checking students' input; methods of automated deduction
applied to prove post-conditions for particular problem solutions; combinations
of deduction and computation enabling systems to propose next steps; automated
provers specific for dynamic geometry systems; proof and proving in mathematics
education.
  ThEdu'18 was a vibrant workshop, with one invited talk and six contributions.
It triggered the post-proceedings at hand.
Deep, spatially coherent Occupancy Maps based on Radar Measurements
Authors: Daniel Bauer, Lars Kuhnert, Lutz Eckstein
Date: 2019-03-29 00:00:00

One essential step to realize modern driver assistance technology is the
accurate knowledge about the location of static objects in the environment. In
this work, we use artificial neural networks to predict the occupation state of
a whole scene in an end-to-end manner. This stands in contrast to the
traditional approach of accumulating each detection's influence on the
occupancy state and allows to learn spatial priors which can be used to
interpolate the environment's occupancy state. We show that these priors make
our method suitable to predict dense occupancy estimations from sparse, highly
uncertain inputs, as given by automotive radars, even for complex urban
scenarios. Furthermore, we demonstrate that these estimations can be used for
large-scale mapping applications.
A Provable Defense for Deep Residual Networks
Authors: Matthew Mirman, Gagandeep Singh, Martin Vechev
Date: 2019-03-29 00:00:00
Tasks: Adversarial Defense, Novel Concepts

We present a training system, which can provably defend significantly larger neural networks than previously possible, including ResNet-34 and DenseNet-100. Our approach is based on differentiable abstract interpretation and introduces two novel concepts: (i) abstract layers for fine-tuning the precision and scalability of the abstraction, (ii) a flexible domain specific language (DSL) for describing training objectives that combine abstract and concrete losses with arbitrary specifications. Our training method is implemented in the DiffAI system.
Probabilistic Forecasting of Sensory Data with Generative Adversarial Networks - ForGAN
Authors: Alireza Koochali, Peter Schichtel, Sheraz Ahmed, Andreas Dengel
Date: 2019-03-29 00:00:00
Tasks: Generative Adversarial Network, Probabilistic Time Series Forecasting, regression, Time Series, Time Series Analysis, Time Series Forecasting, Univariate Time Series Forecasting

Time series forecasting is one of the challenging problems for humankind.
Traditional forecasting methods using mean regression models have severe
shortcomings in reflecting real-world fluctuations. While new probabilistic
methods rush to rescue, they fight with technical difficulties like quantile
crossing or selecting a prior distribution. To meld the different strengths of
these fields while avoiding their weaknesses as well as to push the boundary of
the state-of-the-art, we introduce ForGAN - one step ahead probabilistic
forecasting with generative adversarial networks. ForGAN utilizes the power of
the conditional generative adversarial network to learn the data generating
distribution and compute probabilistic forecasts from it. We argue how to
evaluate ForGAN in opposition to regression methods. To investigate
probabilistic forecasting of ForGAN, we create a new dataset and demonstrate
our method abilities on it. This dataset will be made publicly available for
comparison. Furthermore, we test ForGAN on two publicly available datasets,
namely Mackey-Glass dataset and Internet traffic dataset (A5M) where the
impressive performance of ForGAN demonstrate its high capability in forecasting
future values.
Infinite Brain MR Images: PGGAN-based Data Augmentation for Tumor Detection
Authors: Changhee Han, Leonardo Rundo, Ryosuke Araki, Yujiro Furukawa, Giancarlo Mauri, Hideki Nakayama, Hideaki Hayashi
Date: 2019-03-29 00:00:00
Tasks: Data Augmentation

Due to the lack of available annotated medical images, accurate
computer-assisted diagnosis requires intensive Data Augmentation (DA)
techniques, such as geometric/intensity transformations of original images;
however, those transformed images intrinsically have a similar distribution to
the original ones, leading to limited performance improvement. To fill the data
lack in the real image distribution, we synthesize brain contrast-enhanced
Magnetic Resonance (MR) images---realistic but completely different from the
original ones---using Generative Adversarial Networks (GANs). This study
exploits Progressive Growing of GANs (PGGANs), a multi-stage generative
training method, to generate original-sized 256 X 256 MR images for
Convolutional Neural Network-based brain tumor detection, which is challenging
via conventional GANs; difficulties arise due to unstable GAN training with
high resolution and a variety of tumors in size, location, shape, and contrast.
Our preliminary results show that this novel PGGAN-based DA method can achieve
promising performance improvement, when combined with classical DA, in tumor
detection and also in other medical imaging tasks.
CNN-based Prostate Zonal Segmentation on T2-weighted MR Images: A Cross-dataset Study
Authors: Leonardo Rundo, Changhee Han, Jin Zhang, Ryuichiro Hataya, Yudai Nagano, Carmelo Militello, Claudio Ferretti, Marco S. Nobile, Andrea Tangherloni, Maria Carla Gilardi, Salvatore Vitabile, Hideki Nakayama, Giancarlo Mauri
Date: 2019-03-29 00:00:00
Tasks: Segmentation

Prostate cancer is the most common cancer among US men. However, prostate
imaging is still challenging despite the advances in multi-parametric Magnetic
Resonance Imaging (MRI), which provides both morphologic and functional
information pertaining to the pathological regions. Along with whole prostate
gland segmentation, distinguishing between the Central Gland (CG) and
Peripheral Zone (PZ) can guide towards differential diagnosis, since the
frequency and severity of tumors differ in these regions; however, their
boundary is often weak and fuzzy. This work presents a preliminary study on
Deep Learning to automatically delineate the CG and PZ, aiming at evaluating
the generalization ability of Convolutional Neural Networks (CNNs) on two
multi-centric MRI prostate datasets. Especially, we compared three CNN-based
architectures: SegNet, U-Net, and pix2pix. In such a context, the segmentation
performances achieved with/without pre-training were compared in 4-fold
cross-validation. In general, U-Net outperforms the other methods, especially
when training and testing are performed on multiple datasets.
Learning Relational Representations with Auto-encoding Logic Programs
Authors: Sebastijan Dumancic, Tias Guns, Wannes Meert, Hendrik Blockeel
Date: 2019-03-29 00:00:00
Tasks: Deep Learning, Relational Reasoning, Representation Learning

Deep learning methods capable of handling relational data have proliferated over the last years. In contrast to traditional relational learning methods that leverage first-order logic for representing such data, these deep learning methods aim at re-representing symbolic relational data in Euclidean spaces. They offer better scalability, but can only numerically approximate relational structures and are less flexible in terms of reasoning tasks supported. This paper introduces a novel framework for relational representation learning that combines the best of both worlds. This framework, inspired by the auto-encoding principle, uses first-order logic as a data representation language, and the mapping between the original and latent representation is done by means of logic programs instead of neural networks. We show how learning can be cast as a constraint optimisation problem for which existing solvers can be used. The use of logic as a representation language makes the proposed framework more accurate (as the representation is exact, rather than approximate), more flexible, and more interpretable than deep learning methods. We experimentally show that these latent representations are indeed beneficial in relational learning tasks.
Toroidal AutoEncoder
Authors: Maciej Mikulski, Jaroslaw Duda
Date: 2019-03-28 00:00:00

Enforcing distributions of latent variables in neural networks is an active
subject. It is vital in all kinds of generative models, where we want to be
able to interpolate between points in the latent space, or sample from it.
Modern generative AutoEncoders (AE) like WAE, SWAE, CWAE add a regularizer to
the standard (deterministic) AE, which allows to enforce Gaussian distribution
in the latent space. Enforcing different distributions, especially
topologically nontrivial, might bring some new interesting possibilities, but
this subject seems unexplored so far.
  This article proposes a new approach to enforce uniform distribution on
d-dimensional torus. We introduce a circular spring loss, which enforces
minibatch points to be equally spaced and satisfy cyclic boundary conditions.
  As example of application we propose multiple-path morphing. Minimal distance
geodesic between two points in uniform distribution on latent space of angles
becomes a line, however, torus topology allows us to choose such lines in
alternative ways, going through different edges of $[-\pi,\pi]^d$.
  Further applications to explore can be for example trying to learn real-life
topologically nontrivial spaces of features, like rotations to automatically
recognize 2D rotation of an object in picture by training on relative angles,
or even 3D rotations by additionally using spherical features - this way
morphing should be close to object rotation.
An analysis of the cost of hyper-parameter selection via split-sample validation, with applications to penalized regression
Authors: Jean Feng, Noah Simon
Date: 2019-03-28 00:00:00
Tasks: Open-Ended Question Answering, regression

In the regression setting, given a set of hyper-parameters, a
model-estimation procedure constructs a model from training data. The optimal
hyper-parameters that minimize generalization error of the model are usually
unknown. In practice they are often estimated using split-sample validation. Up
to now, there is an open question regarding how the generalization error of the
selected model grows with the number of hyper-parameters to be estimated. To
answer this question, we establish finite-sample oracle inequalities for
selection based on a single training/test split and based on cross-validation.
We show that if the model-estimation procedures are smoothly parameterized by
the hyper-parameters, the error incurred from tuning hyper-parameters shrinks
at nearly a parametric rate. Hence for semi- and non-parametric
model-estimation procedures with a fixed number of hyper-parameters, this
additional error is negligible. For parametric model-estimation procedures,
adding a hyper-parameter is roughly equivalent to adding a parameter to the
model itself. In addition, we specialize these ideas for penalized regression
problems with multiple penalty parameters. We establish that the fitted models
are Lipschitz in the penalty parameters and thus our oracle inequalities apply.
This result encourages development of regularization methods with many penalty
parameters.
Implicit Langevin Algorithms for Sampling From Log-concave Densities
Authors: Liam Hodgkinson, Robert Salomone, Fred Roosta
Date: 2019-03-29 00:00:00

For sampling from a log-concave density, we study implicit integrators resulting from $\theta$-method discretization of the overdamped Langevin diffusion stochastic differential equation. Theoretical and algorithmic properties of the resulting sampling methods for $ \theta \in [0,1] $ and a range of step sizes are established. Our results generalize and extend prior works in several directions. In particular, for $\theta\ge1/2$, we prove geometric ergodicity and stability of the resulting methods for all step sizes. We show that obtaining subsequent samples amounts to solving a strongly-convex optimization problem, which is readily achievable using one of numerous existing methods. Numerical examples supporting our theoretical analysis are also presented.
On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models
Authors: Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, Ying Nian Wu
Date: 2019-03-29 00:00:00
Tasks: Anatomy

This study investigates the effects of Markov chain Monte Carlo (MCMC) sampling in unsupervised Maximum Likelihood (ML) learning. Our attention is restricted to the family of unnormalized probability densities for which the negative log density (or energy function) is a ConvNet. We find that many of the techniques used to stabilize training in previous studies are not necessary. ML learning with a ConvNet potential requires only a few hyper-parameters and no regularization. Using this minimal framework, we identify a variety of ML learning outcomes that depend solely on the implementation of MCMC sampling. On one hand, we show that it is easy to train an energy-based model which can sample realistic images with short-run Langevin. ML can be effective and stable even when MCMC samples have much higher energy than true steady-state samples throughout training. Based on this insight, we introduce an ML method with purely noise-initialized MCMC, high-quality short-run synthesis, and the same budget as ML with informative MCMC initialization such as CD or PCD. Unlike previous models, our energy model can obtain realistic high-diversity samples from a noise signal after training. On the other hand, ConvNet potentials learned with non-convergent MCMC do not have a valid steady-state and cannot be considered approximate unnormalized densities of the training data because long-run MCMC samples differ greatly from observed images. We show that it is much harder to train a ConvNet potential to learn a steady-state over realistic images. To our knowledge, long-run MCMC samples of all previous models lose the realism of short-run samples. With correct tuning of Langevin noise, we train the first ConvNet potentials for which long-run and steady-state MCMC samples are realistic images.
The False Positive Control Lasso
Authors: Erik Drysdale, Yingwei Peng, Timothy P. Hanna, Paul Nguyen, Anna Goldenberg
Date: 2019-03-29 00:00:00

In high dimensional settings where a small number of regressors are expected
to be important, the Lasso estimator can be used to obtain a sparse solution
vector with the expectation that most of the non-zero coefficients are
associated with true signals. While several approaches have been developed to
control the inclusion of false predictors with the Lasso, these approaches are
limited by relying on asymptotic theory, having to empirically estimate terms
based on theoretical quantities, assuming a continuous response class with
Gaussian noise and design matrices, or high computation costs. In this paper we
show how: (1) an existing model (the SQRT-Lasso) can be recast as a method of
controlling the number of expected false positives, (2) how a similar estimator
can used for all other generalized linear model classes, and (3) this approach
can be fit with existing fast Lasso optimization solvers. Our justification for
false positive control using randomly weighted self-normalized sum theory is to
our knowledge novel. Moreover, our estimator's properties hold in finite
samples up to some approximation error which we find in practical settings to
be negligible under a strict mutual incoherence condition.
A proof of convergence of multi-class logistic regression network
Authors: Marek Rychlik
Date: 2019-03-29 00:00:00
Tasks: regression

This paper revisits the special type of a neural network known under two names. In the statistics and machine learning community it is known as a multi-class logistic regression neural network. In the neural network community, it is simply the soft-max layer. The importance is underscored by its role in deep learning: as the last layer, whose autput is actually the classification of the input patterns, such as images. Our exposition focuses on mathematically rigorous derivation of the key equation expressing the gradient. The fringe benefit of our approach is a fully vectorized expression, which is a basis of an efficient implementation. The second result of this paper is the positivity of the second derivative of the cross-entropy loss function as function of the weights. This result proves that optimization methods based on convexity may be used to train this network. As a corollary, we demonstrate that no $L^2$-regularizer is needed to guarantee convergence of gradient descent.