pwc-archive/papers-with-abstracts huggingface dataset:
The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares
Authors: Rong Ge, Sham M. Kakade, Rahul Kidambi, Praneeth Netrapalli
Date: 2019-04-29 00:00:00
Proceeding: NeurIPS 2019 12
Tasks: Stochastic Optimization

Minimax optimal convergence rates for classes of stochastic convex optimization problems are well characterized, where the majority of results utilize iterate averaged stochastic gradient descent (SGD) with polynomially decaying step sizes. In contrast, SGD's final iterate behavior has received much less attention despite their widespread use in practice. Motivated by this observation, this work provides a detailed study of the following question: what rate is achievable using the final iterate of SGD for the streaming least squares regression problem with and without strong convexity? First, this work shows that even if the time horizon T (i.e. the number of iterations SGD is run for) is known in advance, SGD's final iterate behavior with any polynomially decaying learning rate scheme is highly sub-optimal compared to the minimax rate (by a condition number factor in the strongly convex case and a factor of $\sqrt{T}$ in the non-strongly convex case). In contrast, this paper shows that Step Decay schedules, which cut the learning rate by a constant factor every constant number of epochs (i.e., the learning rate decays geometrically) offers significant improvements over any polynomially decaying step sizes. In particular, the final iterate behavior with a step decay schedule is off the minimax rate by only $log$ factors (in the condition number for strongly convex case, and in T for the non-strongly convex case). Finally, in stark contrast to the known horizon case, this paper shows that the anytime (i.e. the limiting) behavior of SGD's final iterate is poor (in that it queries iterates with highly sub-optimal function value infinitely often, i.e. in a limsup sense) irrespective of the stepsizes employed. These results demonstrate the subtlety in establishing optimal learning rate schemes (for the final iterate) for stochastic gradient procedures in fixed time horizon settings.
Incorporating prior financial domain knowledge into neural networks for implied volatility surface prediction
Authors: Yu Zheng, Yongxin Yang, Bo-Wei Chen
Date: 2019-04-29 00:00:00

In this paper we develop a novel neural network model for predicting implied volatility surface. Prior financial domain knowledge is taken into account. A new activation function that incorporates volatility smile is proposed, which is used for the hidden nodes that process the underlying asset price. In addition, financial conditions, such as the absence of arbitrage, the boundaries and the asymptotic slope, are embedded into the loss function. This is one of the very first studies which discuss a methodological framework that incorporates prior financial domain knowledge into neural network architecture design and model training. The proposed model outperforms the benchmarked models with the option data on the S&P 500 index over 20 years. More importantly, the domain knowledge is satisfied empirically, showing the model is consistent with the existing financial theories and conditions related to implied volatility surface.
Wave Physics as an Analog Recurrent Neural Network
Authors: Tyler W. Hughes, Ian A. D. Williamson, Momchil Minkov, Shanhui Fan
Date: 2019-04-29 00:00:00
Tasks: BIG-bench Machine Learning, Vowel Classification

Analog machine learning hardware platforms promise to be faster and more energy-efficient than their digital counterparts. Wave physics, as found in acoustics and optics, is a natural candidate for building analog processors for time-varying signals. Here we identify a mapping between the dynamics of wave physics, and the computation in recurrent neural networks. This mapping indicates that physical wave systems can be trained to learn complex features in temporal data, using standard training techniques for neural networks. As a demonstration, we show that an inverse-designed inhomogeneous medium can perform vowel classification on raw audio signals as their waveforms scatter and propagate through it, achieving performance comparable to a standard digital implementation of a recurrent neural network. These findings pave the way for a new class of analog machine learning platforms, capable of fast and efficient processing of information in its native domain.
Relational Collaborative Filtering:Modeling Multiple Item Relations for Recommendation
Authors: Xin Xin, Xiangnan He, Yongfeng Zhang, Yongdong Zhang, Joemon Jose
Date: 2019-04-29 00:00:00
Tasks: Collaborative Filtering, Recommendation Systems, Relation

Existing item-based collaborative filtering (ICF) methods leverage only the relation of collaborative similarity. Nevertheless, there exist multiple relations between items in real-world scenarios. Distinct from the collaborative similarity that implies co-interact patterns from the user perspective, these relations reveal fine-grained knowledge on items from different perspectives of meta-data, functionality, etc. However, how to incorporate multiple item relations is less explored in recommendation research. In this work, we propose Relational Collaborative Filtering (RCF), a general framework to exploit multiple relations between items in recommender system. We find that both the relation type and the relation value are crucial in inferring user preference. To this end, we develop a two-level hierarchical attention mechanism to model user preference. The first-level attention discriminates which types of relations are more important, and the second-level attention considers the specific relation values to estimate the contribution of a historical item in recommending the target item. To make the item embeddings be reflective of the relational structure between items, we further formulate a task to preserve the item relations, and jointly train it with the recommendation task of preference modeling. Empirical results on two real datasets demonstrate the strong performance of RCF. Furthermore, we also conduct qualitative analyses to show the benefits of explanations brought by the modeling of multiple item relations.
TileGAN: Synthesis of Large-Scale Non-Homogeneous Textures
Authors: Anna Frühstück, Ibraheem Alhashim, Peter Wonka
Date: 2019-04-29 00:00:00
Tasks: Image Generation, Image Stylization, Texture Synthesis

We tackle the problem of texture synthesis in the setting where many input
images are given and a large-scale output is required. We build on recent
generative adversarial networks and propose two extensions in this paper.
First, we propose an algorithm to combine outputs of GANs trained on a smaller
resolution to produce a large-scale plausible texture map with virtually no
boundary artifacts. Second, we propose a user interface to enable artistic
control. Our quantitative and qualitative results showcase the generation of
synthesized high-resolution maps consisting of up to hundreds of megapixels as
a case in point.
Constraint-Aware Neural Networks for Riemann Problems
Authors: Jim Magiera, Deep Ray, Jan S. Hesthaven, Christian Rohde
Date: 2019-04-29 00:00:00

Neural networks are increasingly used in complex (data-driven) simulations as
surrogates or for accelerating the computation of classical surrogates. In many
applications physical constraints, such as mass or energy conservation, must be
satisfied to obtain reliable results. However, standard machine learning
algorithms are generally not tailored to respect such constraints. We propose
two different strategies to generate constraint-aware neural networks. We test
their performance in the context of front-capturing schemes for strongly
nonlinear wave motion in compressible fluid flow. Precisely, in this context
so-called Riemann problems have to be solved as surrogates. Their solution
describes the local dynamics of the captured wave front in numerical
simulations. Three model problems are considered: a cubic flux model problem,
an isothermal two-phase flow model, and the Euler equations. We demonstrate
that a decrease in the constraint deviation correlates with low discretization
errors for all model problems, in addition to the structural advantage of
fulfilling the constraint.
Graph Matching Networks for Learning the Similarity of Graph Structured Objects
Authors: Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, Pushmeet Kohli
Date: 2019-04-29 00:00:00
Proceeding: ICLR 2019 5
Tasks: Graph Attention, Graph Matching, Retrieval

This paper addresses the challenging problem of retrieval and matching of graph structured objects, and makes two key contributions. First, we demonstrate how Graph Neural Networks (GNN), which have emerged as an effective model for various supervised prediction problems defined on structured data, can be trained to produce embedding of graphs in vector spaces that enables efficient similarity reasoning. Second, we propose a novel Graph Matching Network model that, given a pair of graphs as input, computes a similarity score between them by jointly reasoning on the pair through a new cross-graph attention-based matching mechanism. We demonstrate the effectiveness of our models on different domains including the challenging problem of control-flow-graph based function similarity search that plays an important role in the detection of vulnerabilities in software systems. The experimental analysis demonstrates that our models are not only able to exploit structure in the context of similarity learning but they can also outperform domain-specific baseline systems that have been carefully hand-engineered for these problems.
Routing Networks and the Challenges of Modular and Compositional Computation
Authors: Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, Tim Klinger
Date: 2019-04-29 00:00:00
Tasks: Language Modeling, Language Modelling, Multi-Task Learning, Question Answering, Reading Comprehension, Visual Question Answering, Visual Question Answering (VQA)

Compositionality is a key strategy for addressing combinatorial complexity
and the curse of dimensionality. Recent work has shown that compositional
solutions can be learned and offer substantial gains across a variety of
domains, including multi-task learning, language modeling, visual question
answering, machine comprehension, and others. However, such models present
unique challenges during training when both the module parameters and their
composition must be learned jointly. In this paper, we identify several of
these issues and analyze their underlying causes. Our discussion focuses on
routing networks, a general approach to this problem, and examines empirically
the interplay of these challenges and a variety of design decisions. In
particular, we consider the effect of how the algorithm decides on module
composition, how the algorithm updates the modules, and if the algorithm uses
regularization.
Free Gap Information from the Differentially Private Sparse Vector and Noisy Max Mechanisms
Authors: Zeyu Ding, Yuxin Wang, Danfeng Zhang, Daniel Kifer
Date: 2019-04-29 00:00:00

Noisy Max and Sparse Vector are selection algorithms for differential privacy and serve as building blocks for more complex algorithms. In this paper we show that both algorithms can release additional information for free (i.e., at no additional privacy cost). Noisy Max is used to return the approximate maximizer among a set of queries. We show that it can also release for free the noisy gap between the approximate maximizer and runner-up. This free information can improve the accuracy of certain subsequent counting queries by up to 50%. Sparse Vector is used to return a set of queries that are approximately larger than a fixed threshold. We show that it can adaptively control its privacy budget (use less budget for queries that are likely to be much larger than the threshold) in order to increase the amount of queries it can process. These results follow from a careful privacy analysis.
A Review of Modularization Techniques in Artificial Neural Networks
Authors: Mohammed Amer, Tomás Maul
Date: 2019-04-29 00:00:00

Artificial neural networks (ANNs) have achieved significant success in
tackling classical and modern machine learning problems. As learning problems
grow in scale and complexity, and expand into multi-disciplinary territory, a
more modular approach for scaling ANNs will be needed. Modular neural networks
(MNNs) are neural networks that embody the concepts and principles of
modularity. MNNs adopt a large number of different techniques for achieving
modularization. Previous surveys of modularization techniques are relatively
scarce in their systematic analysis of MNNs, focusing mostly on empirical
comparisons and lacking an extensive taxonomical framework. In this review, we
aim to establish a solid taxonomy that captures the essential properties and
relationships of the different variants of MNNs. Based on an investigation of
the different levels at which modularization techniques act, we attempt to
provide a universal and systematic framework for theorists studying MNNs, also
trying along the way to emphasise the strengths and weaknesses of different
modularization approaches in order to highlight good practices for neural
network practitioners.
Localization, Detection and Tracking of Multiple Moving Sound Sources with a Convolutional Recurrent Neural Network
Authors: Sharath Adavanne, Archontis Politis, Tuomas Virtanen
Date: 2019-04-29 00:00:00

This paper investigates the joint localization, detection, and tracking of
sound events using a convolutional recurrent neural network (CRNN). We use a
CRNN previously proposed for the localization and detection of stationary
sources, and show that the recurrent layers enable the spatial tracking of
moving sources when trained with dynamic scenes. The tracking performance of
the CRNN is compared with a stand-alone tracking method that combines a
multi-source (DOA) estimator and a particle filter. Their respective
performance is evaluated in various acoustic conditions such as anechoic and
reverberant scenarios, stationary and moving sources at several angular
velocities, and with a varying number of overlapping sources. The results show
that the CRNN manages to track multiple sources more consistently than the
parametric method across acoustic scenarios, but at the cost of higher
localization error.
Competitive Statistical Estimation with Strategic Data Sources
Authors: Tyler Westenbroek, Roy Dong, Lillian J. Ratliff, S. Shankar Sastry
Date: 2019-04-29 00:00:00

In recent years, data has played an increasingly important role in the
economy as a good in its own right. In many settings, data aggregators cannot
directly verify the quality of the data they purchase, nor the effort exerted
by data sources when creating the data. Recent work has explored mechanisms to
ensure that the data sources share high quality data with a single data
aggregator, addressing the issue of moral hazard. Oftentimes, there is a
unique, socially efficient solution.
  In this paper, we consider data markets where there is more than one data
aggregator. Since data can be cheaply reproduced and transmitted once created,
data sources may share the same data with more than one aggregator, leading to
free-riding between data aggregators. This coupling can lead to non-uniqueness
of equilibria and social inefficiency. We examine a particular class of
mechanisms that have received study recently in the literature, and we
characterize all the generalized Nash equilibria of the resulting data market.
We show that, in contrast to the single-aggregator case, there is either
infinitely many generalized Nash equilibria or none. We also provide necessary
and sufficient conditions for all equilibria to be socially inefficient. In our
analysis, we identify the components of these mechanisms which give rise to
these undesirable outcomes, showing the need for research into mechanisms for
competitive settings with multiple data purchasers and sellers.
Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
Authors: Xin Chen, Lingxi Xie, Jun Wu, Qi Tian
Date: 2019-04-29 00:00:00
Proceeding: ICCV 2019 10
Tasks: GPU, Neural Architecture Search

Recently, differentiable search methods have made major progress in reducing
the computational costs of neural architecture search. However, these
approaches often report lower accuracy in evaluating the searched architecture
or transferring it to another dataset. This is arguably due to the large gap
between the architecture depths in search and evaluation scenarios. In this
paper, we present an efficient algorithm which allows the depth of searched
architectures to grow gradually during the training procedure. This brings two
issues, namely, heavier computational overheads and weaker search stability,
which we solve using search space approximation and regularization,
respectively. With a significantly reduced search time (~7 hours on a single
GPU), our approach achieves state-of-the-art performance on both the proxy
dataset (CIFAR10 or CIFAR100) and the target dataset (ImageNet). Code is
available at https://github.com/chenxin061/pdarts.
DAC: The Double Actor-Critic Architecture for Learning Options
Authors: Shangtong Zhang, Shimon Whiteson
Date: 2019-04-29 00:00:00
Proceeding: NeurIPS 2019 12
Tasks: Transfer Learning

We reformulate the option framework as two parallel augmented MDPs. Under this novel formulation, all policy optimization algorithms can be used off the shelf to learn intra-option policies, option termination conditions, and a master policy over options. We apply an actor-critic algorithm on each augmented MDP, yielding the Double Actor-Critic (DAC) architecture. Furthermore, we show that, when state-value functions are used as critics, one critic can be expressed in terms of the other, and hence only one critic is necessary. We conduct an empirical study on challenging robot simulation tasks. In a transfer learning setting, DAC outperforms both its hierarchy-free counterpart and previous gradient-based option learning algorithms.
End-to-end Cloud Segmentation in High-Resolution Multispectral Satellite Imagery Using Deep Learning
Authors: Giorgio Morales, Alejandro Ramírez, Joel Telles
Date: 2019-04-29 00:00:00
Tasks: Specificity

Segmenting clouds in high-resolution satellite images is an arduous and
challenging task due to the many types of geographies and clouds a satellite
can capture. Therefore, it needs to be automated and optimized, specially for
those who regularly process great amounts of satellite images, such as
governmental institutions. In that sense, the contribution of this work is
twofold: We present the CloudPeru2 dataset, consisting of 22,400 images of
512x512 pixels and their respective hand-drawn cloud masks, as well as the
proposal of an end-to-end segmentation method for clouds using a Convolutional
Neural Network (CNN) based on the Deeplab v3+ architecture. The results over
the test set achieved an accuracy of 96.62%, precision of 96.46%, specificity
of 98.53%, and sensitivity of 96.72% which is superior to the compared methods.
DeepHMap++: Combined Projection Grouping and Correspondence Learning for Full DoF Pose Estimation
Authors: Mingliang Fu, Weijia Zhou
Date: 2019-04-29 00:00:00
Tasks: Object, Pose Estimation

In recent years, estimating the 6D pose of object instances with
convolutional neural network (CNN) has received considerable attention.
Depending on whether intermediate cues are used, the relevant literature can be
roughly divided into two broad categories: direct methods and two stage
pipelines. For the latter, intermediate cues, such as 3D object coordinates,
semantic keypoints, or virtual control points instead of pose parameters are
regressed by CNN in the first stage. Object pose can then be solved by
correspondence constraints constructed with these intermediate cues. In this
paper, we focus on the postprocessing of a two-stage pipeline and propose to
combine two learning concepts for estimating object pose under challenging
scenes: projection grouping on one side, and correspondence learning on the
other. We firstly employ a local patch based method to predict projection
heatmaps which denote the confidence distribution of projection of 3D bounding
box's corners. A projection grouping module is then proposed to remove
redundant local maxima from each layer of heatmaps. Instead of directly feeding
2D-3D correspondences to the perspective-n-point (PnP) algorithm, multiple
correspondence hypotheses are sampled from local maxima and its corresponding
neighborhood and ranked by a correspondence-evaluation network. Finally,
correspondences with higher confidence are selected to determine object pose.
Extensive experiments on three public datasets demonstrate that the proposed
framework outperforms several state of the art methods.
Solo or Ensemble? Choosing a CNN Architecture for Melanoma Classification
Authors: Fábio Perez, Sandra Avila, Eduardo Valle
Date: 2019-04-29 00:00:00
Tasks: Classification, General Classification, Medical Image Analysis, Skin Lesion Classification, Skin Lesion Identification, Transfer Learning

Convolutional neural networks (CNNs) deliver exceptional results for computer
vision, including medical image analysis. With the growing number of available
architectures, picking one over another is far from obvious. Existing art
suggests that, when performing transfer learning, the performance of CNN
architectures on ImageNet correlates strongly with their performance on target
tasks. We evaluate that claim for melanoma classification, over 9 CNNs
architectures, in 5 sets of splits created on the ISIC Challenge 2017 dataset,
and 3 repeated measures, resulting in 135 models. The correlations we found
were, to begin with, much smaller than those reported by existing art, and
disappeared altogether when we considered only the top-performing networks:
uncontrolled nuisances (i.e., splits and randomness) overcome any of the
analyzed factors. Whenever possible, the best approach for melanoma
classification is still to create ensembles of multiple models. We compared two
choices for selecting which models to ensemble: picking them at random (among a
pool of high-quality ones) vs. using the validation set to determine which ones
to pick first. For small ensembles, we found a slight advantage on the second
approach but found that random choice was also competitive. Although our aim in
this paper was not to maximize performance, we easily reached AUCs comparable
to the first place on the ISIC Challenge 2017.
Globally optimal vertical direction estimation in Atlanta World
Authors: Yinlong Liu, Alois Knoll, Guang Chen
Date: 2019-04-29 00:00:00

In man-made environments, such as indoor and urban scenes, most of the objects and structures are organized in the form of orthogonal and parallel planes. These planes can be approximated by the Atlanta world assumption, in which the normals of planes can be represented by the Atlanta frames. Atlanta world assumption, which can be considered as a generalized Manhattan world assumption, has one vertical frame and multiple horizontal frames. Conventionally, given a set of inputs such as surface normals, the Atlanta frame estimation problem can be solved in one-time by branch-and-bound (BnB). However, the runtime of the BnB algorithm will increase greatly when the dimensionality (i.e., the number of horizontal frames) increases. In this paper, we estimate only the vertical direction instead of all Atlanta frames at once. Accordingly, we propose a vertical direction estimation method by considering the relationship between the vertical frame and horizontal frames. Concretely, our approach employs a BnB algorithm to search the vertical direction guaranteeing global optimality without requiring prior knowledge of the number of Atlanta frames. Four novel bounds by mapping 3D-hemisphere to a 2D region are investigated to guarantee convergence. We verify the validity of the proposed method in various challenging synthetic and real-world data.