pwc-archive/papers-with-abstracts huggingface dataset:
Back to Simplicity: How to Train Accurate BNNs from Scratch?
Authors: Joseph Bethge, Haojin Yang, Marvin Bornstein, Christoph Meinel
Date: 2019-06-19 00:00:00
Tasks: Quantization

Binary Neural Networks (BNNs) show promising progress in reducing computational and memory costs but suffer from substantial accuracy degradation compared to their real-valued counterparts on large-scale datasets, e.g., ImageNet. Previous work mainly focused on reducing quantization errors of weights and activations, whereby a series of approximation methods and sophisticated training tricks have been proposed. In this work, we make several observations that challenge conventional wisdom. We revisit some commonly used techniques, such as scaling factors and custom gradients, and show that these methods are not crucial in training well-performing BNNs. On the contrary, we suggest several design principles for BNNs based on the insights learned and demonstrate that highly accurate BNNs can be trained from scratch with a simple training strategy. We propose a new BNN architecture BinaryDenseNet, which significantly surpasses all existing 1-bit CNNs on ImageNet without tricks. In our experiments, BinaryDenseNet achieves 18.6% and 7.6% relative improvement over the well-known XNOR-Network and the current state-of-the-art Bi-Real Net in terms of top-1 accuracy on ImageNet, respectively.
Learning Generalized Transformation Equivariant Representations via Autoencoding Transformations
Authors: Guo-Jun Qi, Liheng Zhang, Xiao Wang
Date: 2019-06-19 00:00:00
Tasks: Translation

Transformation Equivariant Representations (TERs) aim to capture the intrinsic visual structures that equivary to various transformations by expanding the notion of {\em translation} equivariance underlying the success of Convolutional Neural Networks (CNNs). For this purpose, we present both deterministic AutoEncoding Transformations (AET) and probabilistic AutoEncoding Variational Transformations (AVT) models to learn visual representations from generic groups of transformations. While the AET is trained by directly decoding the transformations from the learned representations, the AVT is trained by maximizing the joint mutual information between the learned representation and transformations. This results in Generalized TERs (GTERs) equivariant against transformations in a more general fashion by capturing complex patterns of visual structures beyond the conventional linear equivariance under a transformation group. The presented approach can be extended to (semi-)supervised models by jointly maximizing the mutual information of the learned representation with both labels and transformations. Experiments demonstrate the proposed models outperform the state-of-the-art models in both unsupervised and (semi-)supervised tasks.
Safe and Near-Optimal Policy Learning for Model Predictive Control using Primal-Dual Neural Networks
Authors: Xiaojing Zhang, Monimoy Bujarbaruah, Francesco Borrelli
Date: 2019-06-19 00:00:00
Tasks: Model Predictive Control

In this paper, we propose a novel framework for approximating the explicit MPC law for linear parameter-varying systems using supervised learning. In contrast to most existing approaches, we not only learn the control policy, but also a "certificate policy", that allows us to estimate the sub-optimality of the learned control policy online, during execution-time. We learn both these policies from data using supervised learning techniques, and also provide a randomized method that allows us to guarantee the quality of each learned policy, measured in terms of feasibility and optimality. This in turn allows us to bound the probability of the learned control policy of being infeasible or suboptimal, where the check is performed by the certificate policy. Since our algorithm does not require the solution of an optimization problem during run-time, it can be deployed even on resource-constrained systems. We illustrate the efficacy of the proposed framework on a vehicle dynamics control problem where we demonstrate a speedup of up to two orders of magnitude compared to online optimization with minimal performance degradation.
Steinhaus Filtration and Stable Paths in the Mapper
Authors: Dustin L. Arendt, Matthew Broussard, Bala Krishnamoorthy, Nathaniel Saul, Amber Thrall
Date: 2019-06-19 00:00:00
Tasks: Recommendation Systems, Topological Data Analysis

We define a new filtration called the Steinhaus filtration built from a single cover based on a generalized Steinhaus distance, a generalization of Jaccard distance. The homology persistence module of a Steinhaus filtration with infinitely many cover elements may not be $q$-tame, even when the covers are in a totally bounded space. While this may pose a challenge to derive stability results, we show that the Steinhaus filtration is stable when the cover is finite. We show that while the \v{C}ech and Steinhaus filtrations are not isomorphic in general, they are isomorphic for a finite point set in dimension one. Furthermore, the VR filtration completely determines the $1$-skeleton of the Steinhaus filtration in arbitrary dimension. We then develop a language and theory for stable paths within the Steinhaus filtration. We demonstrate how the framework can be applied to several applications where a standard metric may not be defined but a cover is readily available. We introduce a new perspective for modeling recommendation system datasets. As an example, we look at a movies dataset and we find the stable paths identified in our framework represent a sequence of movies constituting a gentle transition and ordering from one genre to another. For explainable machine learning, we apply the Mapper algorithm for model induction by building a filtration from a single Mapper complex, and provide explanations in the form of stable paths between subpopulations. For illustration, we build a Mapper complex from a supervised machine learning model trained on the FashionMNIST dataset. Stable paths in the Steinhaus filtration provide improved explanations of relationships between subpopulations of images.
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
Authors: Sebastian Goldt, Madhu S. Advani, Andrew M. Saxe, Florent Krzakala, Lenka Zdeborová
Date: 2019-06-18 00:00:00
Proceeding: NeurIPS 2019 12

Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher-student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.
Doubly Robust Crowdsourcing
Authors: Chong Liu, Yu-Xiang Wang
Date: 2019-06-08 00:00:00

Large-scale labeled dataset is the indispensable fuel that ignites the AI revolution as we see today. Most such datasets are constructed using crowdsourcing services such as Amazon Mechanical Turk which provides noisy labels from non-experts at a fair price. The sheer size of such datasets mandates that it is only feasible to collect a few labels per data point. We formulate the problem of test-time label aggregation as a statistical estimation problem of inferring the expected voting score. By imitating workers with supervised learners and using them in a doubly robust estimation framework, we prove that the variance of estimation can be substantially reduced, even if the learner is a poor approximation. Synthetic and real-world experiments show that by combining the doubly robust approach with adaptive worker/item selection rules, we often need much lower label cost to achieve nearly the same accuracy as in the ideal world where all workers label all data points.
Companion Surface of Danger Cylinder and its Role in Solution Variation of P3P Problem
Authors: Bo wang, Hao Hu, Caixia Zhang
Date: 2019-06-04 00:00:00

Traditionally the danger cylinder is intimately related to the solution stability in P3P problem. In this work, we show that the danger cylinder is also closely related to the multiple-solution phenomenon. More specifically, we show when the optical center lies on the danger cylinder, of the 3 possible P3P solutions, i.e., one double solution, and two other solutions, the optical center of the double solution still lies on the danger cylinder, but the optical centers of the other two solutions no longer lie on the danger cylinder. And when the optical center moves on the danger cylinder, accordingly the optical centers of the two other solutions of the corresponding P3P problem form a new surface, characterized by a polynomial equation of degree 12 in the optical center coordinates, called the Companion Surface of Danger Cylinder (CSDC). That means the danger cylinder always has a companion surface. For the significance of CSDC, we show that when the optical center passes through the CSDC, the number of solutions of P3P problem must change by 2. That means CSDC acts as a delimitating surface of the P3P solution space. These new findings shed some new lights on the P3P multi-solution phenomenon, an important issue in PnP study.
REBEC: Robust Evolutionary-based Calibration Approach for the Numerical Wind Wave Model
Authors: Pavel Vychuzhanin, Nikolay O. Nikitin, Anna V. Kalyuzhnaya
Date: 2019-03-19 00:00:00

The adaptation of numerical wind wave models to the local time-spatial conditions is a problem that can be solved by using various calibration techniques. However, the obtained sets of physical parameters become over-tuned to specific events if there is a lack of observations. In this paper, we propose a robust evolutionary calibration approach that allows to build the stochastic ensemble of perturbed models and use it to achieve the trade-off between quality and robustness of the target model. The implemented robust ensemble-based evolutionary calibration (REBEC) approach was compared to the baseline SPEA2 algorithm in a set of experiments with the SWAN wind wave model configuration for the Kara Sea domain. Provided metrics for the set of scenarios confirm the effectiveness of the REBEC approach for the majority of calibration scenarios.
Machine Learning Allows Calibration Models to Predict Trace Element Concentration in Soil with Generalized LIBS Spectra
Authors: Chen Sun, Ye Tian, Liang Gao, YiShuai Niu, Tianlong Zhang, Hua Li, Yuqing Zhang, Zengqi Yue, Nicole Delepine-Gilon, Jin Yu
Date: 2019-02-13 00:00:00
Tasks: BIG-bench Machine Learning, feature selection

Calibration models have been developed for determination of trace elements, silver for instance, in soil using laser-induced breakdown spectroscopy (LIBS). The major concern is the matrix effect. Although it affects the accuracy of LIBS measurements in a general way, the effect appears accentuated for soil because of large variation of chemical and physical properties among different soils. The purpose is to reduce its influence in such way an accurate and soil-independent calibration model can be constructed. At the same time, the developed model should efficiently reduce experimental fluctuations affecting measurement precision. A univariate model first reveals obvious influence of matrix effect and important experimental fluctuation. A multivariate model has been then developed. A key point is the introduction of generalized spectrum where variables representing the soil type are explicitly included. Machine learning has been used to develop the model. After a necessary pretreatment where a feature selection process reduces the dimension of raw spectrum accordingly to the number of available spectra, the data have been fed in to a back-propagation neuronal networks (BPNN) to train and validate the model. The resulted soilindependent calibration model allows average relative error of calibration (REC) and average relative error of prediction (REP) within the range of 5-6%.
Google Vizier: A Service for Black-Box Optimization
Authors: Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, D. Sculley
Date: 2017-01-01 00:00:00
Proceeding: ACM (2017) 2017 1
Tasks: AutoML, BIG-bench Machine Learning, Hyperparameter Optimization, Transfer Learning

Any sufficiently complex system acts as a black box when it becomes easier to experiment with than to understand. Hence, black-box optimization has become increasingly important as systems have become more complex. In this paper we describe Google Vizier, a Google-internal service for performing black-box optimization that has become the de facto parameter tuning engine at Google. Google Vizier is used to optimize many of our machine learning models and other systems, and also provides core capabilities to Google's Cloud Machine Learning HyperTune subsystem. We discuss our requirements, infrastructure design, underlying algorithms, and advanced features such as transfer learning and automated early stopping that the service provides.
AMLA: an AutoML frAmework for Neural Network Design
Authors: Purushotham Kamath, Abhishek Singh, Debo Dutta
Date: 2018-01-01 00:00:00
Proceeding: ICML 2018 AutoML Workshop 2018 1
Tasks: AutoML, Hyperparameter Optimization, image-classification, Image Classification, Neural Architecture Search

AMLA is an Automatic Machine Learning frAmework for implementing and deploying neural architecture search algorithms. Neural architecture search algorithms are AutoML algorithms whose goal is to generate optimal neural network structures for a given task. AMLA is designed to deploy these algorithms at scale and allow comparison of the performance of the networks generated by different AutoML algorithms. Its key architectural features are the decoupling of the network generation from the network evaluation, support for network instrumentation, open model specification and a microservices based architecture for deployment at scale. In AMLA, AutoML algorithms and training/evaluation code are written as containerized microservices that can be deployed at scale on a public or private infrastructure. The microservices communicate via well defined interfaces and models are persisted using standard model definition formats, allowing the plug and play of the AutoML algorithms as well as the AI/ML libraries. This makes it easy to prototype, compare, benchmark and deploy autoML algorithms in production. AMLA is currently being used to deploy an AutoML algorithm that generates Convolutional Neural Networks (CNNs) used for image classification.
Circuitscape in Julia: High Performance Connectivity Modelling to Support Conservation Decisions
Authors: Ranjan Anantharaman, Kimberly Hall, Viral Shah, Alan Edelman
Date: 2019-06-08 00:00:00
Tasks: Vocal Bursts Intensity Prediction

Connectivity across landscapes influences a wide range of conservation-relevant ecological processes, including species movements, gene flow, and the spread of wildfire, pests, and diseases. Recent improvements in remote sensing data suggest great potential to advance connectivity models, but computational constraints hinder these advances. To address this challenge, we upgraded the widely-used Circuitscape connectivity package to the high performance Julia programming language. Circuitscape.jl allows users to solve problems faster via improved parallel processing and solvers, and supports applications to larger problems (e.g., datasets with hundreds of millions of cells). We document speed improvements of up to 1800\%. We also demonstrate scaling of problem sizes up to 437 million grid cells. These improvements allow modelers to work with higher resolution data, larger landscapes and perform sensitivity analysis effortlessly. These improvements accelerate the pace of innovation, helping modelers address pressing challenges like species range shifts under climate change. Our collaboration between ecologists and computer scientists has led to the use of connectivity models to inform conservation decisions. Further, these next generation connectivity models will produce results faster, facilitating stronger engagement with decision-makers.
ATM: A distributed, collaborative, scalable system for automated machine learning
Authors: Thomas Swearingen, Will Drevo, Bennett Cyphers, Alfredo Cuesta-Infante, Arun Ross, Kalyan Veeramachaneni
Date: 2017-12-11 00:00:00
Proceeding: 2017 IEEE International Conference on Big Data (Big Data) 2017 12
Tasks: AutoML, BIG-bench Machine Learning, Feature Engineering, Hyperparameter Optimization, Model Selection

In this paper, we present Auto-Tuned Models, or ATM, a distributed, collaborative, scalable system for automated machine learning. Users of ATM can simply upload a dataset, choose a subset of modeling methods, and choose to use ATM’s hybrid Bayesian and multi-armed bandit optimization system. The distributed system works in a load balanced fashion to quickly deliver results in the form of ready-to-predict models, confusion matrices, cross-validation results, and training timings. By automating hyperparameter tuning and model selection, ATM returns the emphasis of the machine learning workflow to its most irreducible part: feature engineering. We demonstrate the usefulness of ATM on 420 datasets from OpenML and train over 3 million classifiers. Our initial results show ATM can beat human-generated solutions for 30% of the datasets, and can do so in 1/100th of the time.
Supplementary Material for Efficient and Robust Automated Machine Learning
Authors: Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, Frank Hutter
Date: 2015-01-01 00:00:00
Proceeding: NIPS 2015 2015 1
Tasks: BIG-bench Machine Learning, Hyperparameter Optimization

Supplementary Material for Efficient and Robust Automated Machine Learning
Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms
Authors: James Bergstra, Dan Yamins, David D. Cox
Date: 2013-12-12 00:00:00
Proceeding: SCIPY 2013 2013 12
Tasks: Bayesian Optimization, BIG-bench Machine Learning, Hyperparameter Optimization, Model Selection

Sequential model-based optimization (also known as Bayesian optimization) is one of the most efficient methods (per function evaluation) of function minimization. This efficiency makes it appropriate for optimizing the hyperparameters of machine learning algorithms that are slow to train. The Hyperopt library provides algorithms and parallelization infrastructure for performing hyperparameter optimization (model selection) in Python. This paper presents an introductory tutorial on the usage of the Hyperopt library, including the description of search spaces, minimization (in serial and parallel), and the analysis of the results collected in the course of minimization. The paper closes with some discussion of ongoing and future work.
Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn
Authors: Brent Komer, James Bergstra, Chris Eliasmith
Date: 2014-01-01 00:00:00
Proceeding: SCIPY 2014 2014 1
Tasks: AutoML, Benchmarking, Hyperparameter Optimization

Hyperopt-sklearn is a new software project that provides automatic algorithm configuration of the Scikit-learn machine learning library. Following Auto-Weka, we take the view that the choice of classifier and even the choice of preprocessing module can be taken together to represent a single large hyperparameter optimization problem. We use Hyperopt to define a search space that encompasses many standard components (e.g. SVM, RF, KNN, PCA, TFIDF) and common patterns of composing them together. We demonstrate, using search algorithms in Hyperopt and standard benchmarking data sets (MNIST, 20-Newsgroups, Convex Shapes), that searching this space is practical and effective. In particular, we improve on best-known scores for the model space for both MNIST and Convex Shapes.
Katib: A Distributed General AutoML Platform on Kubernetes
Authors: Jinan Zhou, Andrey Velichkevich, Kirill Prosvirov, Anubhav Garg, Yuji Oshima, Debo Dutta
Date: 2019-01-01 00:00:00
Proceeding: USENIX Conference on Operational Machine Learning 2019 2019 1
Tasks: AutoML, Hyperparameter Optimization, Management, Neural Architecture Search

Automatic Machine Learning (AutoML) is a powerful mechanism to design and tune models. We present Katib, a scalable Kubernetes-native general AutoML platform that can support a range of AutoML algorithms including both hyper-parameter tuning and neural architecture search. The system is divided into separate components, encapsulated as micro-services. Each micro-service operates within a Kubernetes pod and communicates with others via well-defined APIs, thus allowing flexible management and scalable deployment at a minimal cost. Together with a powerful user interface, Katib provides a universal platform for researchers as well as enterprises to try, compare and deploy their AutoML algorithms, on any Kubernetes platform.
RBFOpt: an open-source library for black-box optimization with costly function evaluations
Authors: Alberto Costa, Giacomo Nannicini
Date: 2015-11-05 00:00:00
Proceeding: Mathematical Programming Computation 2018 2015 11
Tasks: Hyperparameter Optimization, Model Selection

We consider the problem of optimizing an unknown function given as an oracle over a mixed-integer box-constrained set. We assume that the oracle is expensive to evaluate, so that estimating partial derivatives by finite differences is impractical. In the literature, this is typically called a black-box optimization problem with costly evaluation. This paper describes the solution methodology implemented in the open-source library RBFOpt, available on COIN-OR. The algorithm is based on the Radial Basis Function method originally proposed by Gutmann (J Glob Optim 19:201–227, 2001.), which builds and iteratively refines a surrogate model of the unknown objective function. The two main methodological contributions of this paper are an approach to exploit a noisy but less expensive oracle to accelerate convergence to the optimum of the exact oracle, and the introduction of an automatic model selection phase during the optimization process. Numerical experiments show that RBFOpt is highly competitive on a test set of continuous and mixed-integer nonlinear unconstrained problems taken from the literature: it outperforms the open-source solvers included in our comparison by a large amount, and performs slightly better than a commercial solver. Our empirical evaluation provides insight on which parameterizations of the algorithm are the most effective in practice. The software reviewed as part of this submission was given the Digital Object Identifier (DOI).