pwc-archive/papers-with-abstracts huggingface dataset:
Learning Two-layer Neural Networks with Symmetric Inputs
Authors: Rong Ge, Rohith Kuditipudi, Zhize Li, Xiang Wang
Date: 2018-10-16 00:00:00
Proceeding: ICLR 2019 5
Tasks: Vocal Bursts Valence Prediction

We give a new algorithm for learning a two-layer neural network under a
general class of input distributions. Assuming there is a ground-truth
two-layer network $$ y = A \sigma(Wx) + \xi, $$ where $A,W$ are weight
matrices, $\xi$ represents noise, and the number of neurons in the hidden layer
is no larger than the input or output, our algorithm is guaranteed to recover
the parameters $A,W$ of the ground-truth network. The only requirement on the
input $x$ is that it is symmetric, which still allows highly complicated and
structured input.
  Our algorithm is based on the method-of-moments framework and extends several
results in tensor decompositions. We use spectral algorithms to avoid the
complicated non-convex optimization in learning neural networks. Experiments
show that our algorithm can robustly learn the ground-truth neural network with
a small number of samples for many symmetric input distributions.
ProMP: Proximal Meta-Policy Search
Authors: Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, Pieter Abbeel
Date: 2018-10-16 00:00:00
Proceeding: ICLR 2019 5
Tasks: Meta-Learning, Meta Reinforcement Learning, Reinforcement Learning

Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood. Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively. This leads to poor sample-efficiency during meta-training as well as ineffective task identification strategies. This paper provides a theoretical analysis of credit assignment in gradient-based Meta-RL. Building on the gained insights we develop a novel meta-learning algorithm that overcomes both the issue of poor credit assignment and previous difficulties in estimating meta-policy gradients. By controlling the statistical distance of both pre-adaptation and adapted policies during meta-policy search, the proposed algorithm endows efficient and stable meta-learning. Our approach leads to superior pre-adaptation policy behavior and consistently outperforms previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance.
Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks
Authors: Xiaodong Cui, Wei zhang, Zoltán Tüske, Michael Picheny
Date: 2018-10-16 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Evolutionary Algorithms, Language Modeling, Language Modelling, speech-recognition, Speech Recognition

We propose a population-based Evolutionary Stochastic Gradient Descent (ESGD)
framework for optimizing deep neural networks. ESGD combines SGD and
gradient-free evolutionary algorithms as complementary algorithms in one
framework in which the optimization alternates between the SGD step and
evolution step to improve the average fitness of the population. With a
back-off strategy in the SGD step and an elitist strategy in the evolution
step, it guarantees that the best fitness in the population will never degrade.
In addition, individuals in the population optimized with various SGD-based
optimizers using distinct hyper-parameters in the SGD step are considered as
competing species in a coevolution setting such that the complementarity of the
optimizers is also taken into account. The effectiveness of ESGD is
demonstrated across multiple applications including speech recognition, image
recognition and language modeling, using networks with a variety of deep
architectures.
Approximate Fisher Information Matrix to Characterise the Training of Deep Neural Networks
Authors: Zhibin Liao, Tom Drummond, Ian Reid, Gustavo Carneiro
Date: 2018-10-16 00:00:00
Tasks: General Classification, image-classification, Image Classification

In this paper, we introduce a novel methodology for characterising the
performance of deep learning networks (ResNets and DenseNet) with respect to
training convergence and generalisation as a function of mini-batch size and
learning rate for image classification. This methodology is based on novel
measurements derived from the eigenvalues of the approximate Fisher information
matrix, which can be efficiently computed even for high capacity deep models.
Our proposed measurements can help practitioners to monitor and control the
training process (by actively tuning the mini-batch size and learning rate) to
allow for good training convergence and generalisation. Furthermore, the
proposed measurements also allow us to show that it is possible to optimise the
training process with a new dynamic sampling training approach that
continuously and automatically change the mini-batch size and learning rate
during the training process. Finally, we show that the proposed dynamic
sampling training approach has a faster training time and a competitive
classification accuracy compared to the current state of the art.
DN-ResNet: Efficient Deep Residual Network for Image Denoising
Authors: Haoyu Ren, Mostafa El-Khamy, Jungwon Lee
Date: 2018-10-16 00:00:00
Tasks: Computational Efficiency, Deep Learning, Denoising, Image Denoising, Image Enhancement, Image Restoration

A deep learning approach to blind denoising of images without complete
knowledge of the noise statistics is considered. We propose DN-ResNet, which is
a deep convolutional neural network (CNN) consisting of several residual blocks
(ResBlocks). With cascade training, DN-ResNet is more accurate and more
computationally efficient than the state of art denoising networks. An
edge-aware loss function is further utilized in training DN-ResNet, so that the
denoising results have better perceptive quality compared to conventional loss
function. Next, we introduce the depthwise separable DN-ResNet (DS-DN-ResNet)
utilizing the proposed Depthwise Seperable ResBlock (DS-ResBlock) instead of
standard ResBlock, which has much less computational cost. DS-DN-ResNet is
incrementally evolved by replacing the ResBlocks in DN-ResNet by DS-ResBlocks
stage by stage. As a result, high accuracy and good computational efficiency
are achieved concurrently. Whereas previous state of art deep learning methods
focused on denoising either Gaussian or Poisson corrupted images, we consider
denoising images having the more practical Poisson with additive Gaussian noise
as well. The results show that DN-ResNets are more efficient, robust, and
perform better denoising than current state of art deep learning methods, as
well as the popular variants of the BM3D algorithm, in cases of blind and
non-blind denoising of images corrupted with Poisson, Gaussian or
Poisson-Gaussian noise. Our network also works well for other image enhancement
task such as compressed image restoration.
A survey of automatic de-identification of longitudinal clinical narratives
Authors: Vithya Yogarajan, Michael Mayo, Bernhard Pfahringer
Date: 2018-10-16 00:00:00
Tasks: De-identification, Survey

Use of medical data, also known as electronic health records, in research
helps develop and advance medical science. However, protecting patient
confidentiality and identity while using medical data for analysis is crucial.
Medical data can be in the form of tabular structures (i.e. tables), free-form
narratives, and images. This study focuses on medical data in the free form
longitudinal text. De-identification of electronic health records provides the
opportunity to use such data for research without it affecting patient privacy,
and avoids the need for individual patient consent. In recent years there is
increasing interest in developing an accurate, robust and adaptable automatic
de-identification system for electronic health records. This is mainly due to
the dilemma between the availability of an abundance of health data, and the
inability to use such data in research due to legal and ethical restrictions.
De-identification tracks in competitions such as the 2014 i2b2 UTHealth and the
2016 CEGS N-GRID shared tasks have provided a great platform to advance this
area. The primary reasons for this include the open source nature of the
dataset and the fact that raw psychiatric data were used for 2016 competitions.
This study focuses on noticeable trend changes in the techniques used in the
development of automatic de-identification for longitudinal clinical
narratives. More specifically, the shift from using conditional random fields
(CRF) based systems only or rules (regular expressions, dictionary or
combinations) based systems only, to hybrid models (combining CRF and rules),
and more recently to deep learning based systems. We review the literature and
results that arose from the 2014 and the 2016 competitions and discuss the
outcomes of these systems. We also provide a list of research questions that
emerged from this survey.
A Block Coordinate Descent Proximal Method for Simultaneous Filtering and Parameter Estimation
Authors: Ramin Raziperchikolaei, Harish S. Bhat
Date: 2018-10-16 00:00:00
Tasks: parameter estimation

We propose and analyze a block coordinate descent proximal algorithm (BCD-prox) for simultaneous filtering and parameter estimation of ODE models. As we show on ODE systems with up to d=40 dimensions, as compared to state-of-the-art methods, BCD-prox exhibits increased robustness (to noise, parameter initialization, and hyperparameters), decreased training times, and improved accuracy of both filtered states and estimated parameters. We show how BCD-prox can be used with multistep numerical discretizations, and we establish convergence of BCD-prox under hypotheses that include real systems of interest.
Discriminator Rejection Sampling
Authors: Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, Augustus Odena
Date: 2018-10-16 00:00:00
Proceeding: ICLR 2019 5
Tasks: Image Generation

We propose a rejection sampling scheme using the discriminator of a GAN to
approximately correct errors in the GAN generator distribution. We show that
under quite strict assumptions, this will allow us to recover the data
distribution exactly. We then examine where those strict assumptions break down
and design a practical algorithm - called Discriminator Rejection Sampling
(DRS) - that can be used on real data-sets. Finally, we demonstrate the
efficacy of DRS on a mixture of Gaussians and on the SAGAN model,
state-of-the-art in the image generation task at the time of developing this
work. On ImageNet, we train an improved baseline that increases the Inception
Score from 52.52 to 62.36 and reduces the Frechet Inception Distance from 18.65
to 14.79. We then use DRS to further improve on this baseline, improving the
Inception Score to 76.08 and the FID to 13.75.
Discovering Fair Representations in the Data Domain
Authors: Novi Quadrianto, Viktoriia Sharmanska, Oliver Thomas
Date: 2018-10-15 00:00:00
Proceeding: CVPR 2019 6
Tasks: Attribute, Fairness, Translation

Interpretability and fairness are critical in computer vision and machine
learning applications, in particular when dealing with human outcomes, e.g.
inviting or not inviting for a job interview based on application materials
that may include photographs. One promising direction to achieve fairness is by
learning data representations that remove the semantics of protected
characteristics, and are therefore able to mitigate unfair outcomes. All
available models however learn latent embeddings which comes at the cost of
being uninterpretable. We propose to cast this problem as data-to-data
translation, i.e. learning a mapping from an input domain to a fair target
domain, where a fairness definition is being enforced. Here the data domain can
be images, or any tabular data representation. This task would be
straightforward if we had fair target data available, but this is not the case.
To overcome this, we learn a highly unconstrained mapping by exploiting
statistics of residuals - the difference between input data and its translated
version - and the protected characteristics. When applied to the CelebA dataset
of face images with gender attribute as the protected characteristic, our model
enforces equality of opportunity by adjusting the eyes and lips regions.
Intriguingly, on the same dataset we arrive at similar conclusions when using
semantic attribute representations of images for translation. On face images of
the recent DiF dataset, with the same gender attribute, our method adjusts nose
regions. In the Adult income dataset, also with protected gender attribute, our
model achieves equality of opportunity by, among others, obfuscating the wife
and husband relationship. Analyzing those systematic changes will allow us to
scrutinize the interplay of fairness criterion, chosen protected
characteristics, and prediction performance.
Optimally rotated coordinate systems for adaptive least-squares regression on sparse grids
Authors: Bastian Bohn, Michael Griebel, Jens Oettershagen
Date: 2018-10-15 00:00:00
Tasks: regression

For low-dimensional data sets with a large amount of data points, standard
kernel methods are usually not feasible for regression anymore. Besides simple
linear models or involved heuristic deep learning models, grid-based
discretizations of larger (kernel) model classes lead to algorithms, which
naturally scale linearly in the amount of data points. For moderate-dimensional
or high-dimensional regression tasks, these grid-based discretizations suffer
from the curse of dimensionality. Here, sparse grid methods have proven to
circumvent this problem to a large extent. In this context, space- and
dimension-adaptive sparse grids, which can detect and exploit a given low
effective dimensionality of nominally high-dimensional data, are particularly
successful. They nevertheless rely on an axis-aligned structure of the solution
and exhibit issues for data with predominantly skewed and rotated coordinates.
  In this paper we propose a preprocessing approach for these adaptive sparse
grid algorithms that determines an optimized, problem-dependent coordinate
system and, thus, reduces the effective dimensionality of a given data set in
the ANOVA sense. We provide numerical examples on synthetic data as well as
real-world data to show how an adaptive sparse grid least squares algorithm
benefits from our preprocessing method.
Assessing the Contribution of Semantic Congruency to Multisensory Integration and Conflict Resolution
Authors: Di Fu, Pablo Barros, German I. Parisi, Haiyan Wu, Sven Magg, Xun Liu, Stefan Wermter
Date: 2018-10-15 00:00:00

The efficient integration of multisensory observations is a key property of
the brain that yields the robust interaction with the environment. However,
artificial multisensory perception remains an open issue especially in
situations of sensory uncertainty and conflicts. In this work, we extend
previous studies on audio-visual (AV) conflict resolution in complex
environments. In particular, we focus on quantitatively assessing the
contribution of semantic congruency during an AV spatial localization task. In
addition to conflicts in the spatial domain (i.e. spatially misaligned
stimuli), we consider gender-specific conflicts with male and female avatars.
Our results suggest that while semantically related stimuli affect the
magnitude of the visual bias (perceptually shifting the location of the sound
towards a semantically congruent visual cue), humans still strongly rely on
environmental statistics to solve AV conflicts. Together with previously
reported results, this work contributes to a better understanding of how
multisensory integration and conflict resolution can be modelled in artificial
agents and robots operating in real-world environments.
Using Deep Reinforcement Learning for the Continuous Control of Robotic Arms
Authors: Winfried Lötzsch
Date: 2018-10-15 00:00:00
Tasks: continuous-control, Continuous Control, Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Deep reinforcement learning enables algorithms to learn complex behavior,
deal with continuous action spaces and find good strategies in environments
with high dimensional state spaces. With deep reinforcement learning being an
active area of research and many concurrent inventions, we decided to focus on
a relatively simple robotic task to evaluate a set of ideas that might help to
solve recent reinforcement learning problems. We test a newly created
combination of two commonly used reinforcement learning methods, whether it is
able to learn more effectively than a baseline. We also compare different ideas
to preprocess information before it is fed to the reinforcement learning
algorithm. The goal of this strategy is to reduce training time and eventually
help the algorithm to converge. The concluding evaluation proves the general
applicability of the described concepts by testing them using a simulated
environment. These concepts might be reused for future experiments.
Can Euroscepticism Contribute to a European Public Sphere? The Europeanization of Media Discourses about Euroscepticism across Six Countries
Authors: Anamaria Dutceac Segesten, Michael Bossetta
Date: 2018-10-15 00:00:00
Tasks: Articles

This study compares the media discourses about Euroscepticism in 2014 across
six countries (United Kingdom, Ireland, France, Spain, Sweden, and Denmark). We
assess the extent to which the mass media's reporting of Euroscepticism
indicates the Europeanization of public spheres. Using a mixed-methods approach
combining LDA topic modeling and qualitative coding, we find that approximately
70 per cent of print articles mentioning "Euroscepticism" or "Eurosceptic" are
framed in a non-domestic (i.e. European) context. In five of the six cases
studied, articles exhibiting a European context are strikingly similar in
content, with the British case as the exception. However, coverage of British
Euroscepticism drives Europeanization in other Member States. Bivariate
logistic regressions further reveal three macro-level structural variables that
significantly correlate with a Europeanized media discourse: newspaper type
(tabloid or broadsheet), presence of a strong Eurosceptic party, and
relationship to the EU budget (net contributor or receiver of EU funds).
Marrying Universal Dependencies and Universal Morphology
Authors: Arya D. McCarthy, Miikka Silfverberg, Ryan Cotterell, Mans Hulden, David Yarowsky
Date: 2018-10-15 00:00:00
Proceeding: WS 2018 11

The Universal Dependencies (UD) and Universal Morphology (UniMorph) projects
each present schemata for annotating the morphosyntactic details of language.
Each project also provides corpora of annotated text in many languages - UD at
the token level and UniMorph at the type level. As each corpus is built by
different annotators, language-specific decisions hinder the goal of universal
schemata. With compatibility of tags, each project's annotations could be used
to validate the other's. Additionally, the availability of both type- and
token-level resources would be a boon to tasks such as parsing and homograph
disambiguation. To ease this interoperability, we present a deterministic
mapping from Universal Dependencies v2 features into the UniMorph schema. We
validate our approach by lookup in the UniMorph corpora and find a
macro-average of 64.13% recall. We also note incompatibilities due to paucity
of data on either side. Finally, we present a critical evaluation of the
foundations, strengths, and weaknesses of the two annotation projects.
Robust Neural Machine Translation with Joint Textual and Phonetic Embedding
Authors: Hairong Liu, Mingbo Ma, Liang Huang, Hao Xiong, Zhongjun He
Date: 2018-10-15 00:00:00
Proceeding: ACL 2019 7
Tasks: Automatic Speech Recognition (ASR), Machine Translation, NMT, Speech Recognition, Translation

Neural machine translation (NMT) is notoriously sensitive to noises, but noises are almost inevitable in practice. One special kind of noise is the homophone noise, where words are replaced by other words with similar pronunciations. We propose to improve the robustness of NMT to homophone noises by 1) jointly embedding both textual and phonetic information of source sentences, and 2) augmenting the training dataset with homophone noises. Interestingly, to achieve better translation quality and more robustness, we found that most (though not all) weights should be put on the phonetic rather than textual information. Experiments show that our method not only significantly improves the robustness of NMT to homophone noises, but also surprisingly improves the translation quality on some clean test sets.
Optimizing Agent Behavior over Long Time Scales by Transporting Value
Authors: Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, Greg Wayne
Date: 2018-10-15 00:00:00
Tasks: Reinforcement Learning

Humans spend a remarkable fraction of waking life engaged in acts of "mental
time travel". We dwell on our actions in the past and experience satisfaction
or regret. More than merely autobiographical storytelling, we use these event
recollections to change how we will act in similar scenarios in the future.
This process endows us with a computationally important ability to link actions
and consequences across long spans of time, which figures prominently in
addressing the problem of long-term temporal credit assignment; in artificial
intelligence (AI) this is the question of how to evaluate the utility of the
actions within a long-duration behavioral sequence leading to success or
failure in a task. Existing approaches to shorter-term credit assignment in AI
cannot solve tasks with long delays between actions and consequences. Here, we
introduce a new paradigm for reinforcement learning where agents use recall of
specific memories to credit actions from the past, allowing them to solve
problems that are intractable for existing algorithms. This paradigm broadens
the scope of problems that can be investigated in AI and offers a mechanistic
account of behaviors that may inspire computational models in neuroscience,
psychology, and behavioral economics.
Bringing Order to the Cognitive Fallacy Zoo
Authors: Ardavan S. Nobandegani, William Campoli, Thomas R. Shultz
Date: 2018-10-15 00:00:00

In the eyes of a rationalist like Descartes or Spinoza, human reasoning is
flawless, marching toward uncovering ultimate truth. A few centuries later,
however, culminating in the work of Kahneman and Tversky, human reasoning was
portrayed as anything but flawless, filled with numerous misjudgments, biases,
and cognitive fallacies. With further investigations, new cognitive fallacies
continually emerged, leading to a state of affairs which can fairly be
characterized as the cognitive fallacy zoo! In this largely methodological
work, we formally present a principled way to bring order to this zoo. We
introduce the idea of establishing implication relationships (IRs) between
cognitive fallacies, formally characterizing how one fallacy implies another.
IR is analogous to, and partly inspired by, the fundamental concept of
reduction in computational complexity theory. We present several examples of
IRs involving experimentally well-documented cognitive fallacies: base-rate
neglect, availability bias, conjunction fallacy, decoy effect, framing effect,
and Allais paradox. We conclude by discussing how our work: (i) allows for
identifying those pivotal cognitive fallacies whose investigation would be the
most rewarding research agenda, and importantly (ii) permits a systematized,
guided research program on cognitive fallacies, motivating influential
theoretical as well as experimental avenues of future research.