pwc-archive/papers-with-abstracts huggingface dataset:
RoBO: A Flexible and Robust Bayesian Optimization Framework in Python
Authors: Aaron Klein, Stefan Falkner, Numair Mansur, Frank Hutter
Date: 2017-01-01 00:00:00
Proceeding: NIPS 2017 2017 1
Tasks: Bayesian Optimization, Hyperparameter Optimization

Bayesian optimization is a powerful approach for the global derivative-free optimization of non-convex expensive functions. Even though there is a rich literature on Bayesian optimization, the source code of advanced methods is rarely available, making it difficult for practitioners to use them and for researchers to compare to and extend them. The BSD-licensed python package ROBO, released with this paper, tackles these problems by facilitating both ease of use and extensibility. Beyond the standard methods in Bayesian optimization, RoBO offers (to the best of our knowledge) the only available implementations of Bayesian optimization with Bayesian neural networks, multi-task optimization, and fast Bayesian hyperparameter optimization on large datasets (Fabolas).
Sequential Model-Based Optimization for General Algorithm Configuration
Authors: Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown
Date: 2011-01-18 00:00:00
Proceeding: LION 2011 2011 1
Tasks: Hyperparameter Optimization, model

State-of-the-art algorithms for hard computational problems often expose many parameters that can be modified to improve empirical performance. However, manually exploring the resulting combinatorial space of parameter settings is tedious and tends to lead to unsatisfactory outcomes. Recently, automated approaches for solving this algorithm configuration problem have led to substantial improvements in the state of the art for solving various problems. One promising approach constructs explicit regression models to describe the dependence of target algorithm performance on parameter settings; however, this approach has so far been limited to the optimization of few numerical algorithm parameters on single instances. In this paper, we extend this paradigm for the first time to general algorithm configuration problems, allowing many categorical parameters and optimization for sets of instances. We experimentally validate our new algorithm configuration procedure by optimizing a local search and a tree search solver for the propositional satisfiability problem (SAT), as well as the commercial mixed integer programming (MIP) solver CPLEX. In these experiments, our procedure yielded state-of-the-art performance, and in many cases outperformed the previous best configuration approach.
Automating Feature Engineering
Authors: Udayan Khurana, Fatemeh Nargesian, Horst Samulowitz, Elias Khalil, Deepak Turaga
Date: 2016-01-01 00:00:00
Proceeding: NIPS 2016 2016 1
Tasks: Automated Feature Engineering, Feature Engineering

Feature Engineering is the task of transforming the feature space in a given learning problem to improve the performance of a trained model. It is a crucial but time intensive and skillful process, involving a data scientist or a domain expert. It is often the key determinant of the time and cost required to build an effective learner. In this paper, we discuss our system for performing feature engineering in an automated manner using a combination of exploratory and learning techniques. We also mention our larger charter of an automated data science pipeline.
ExploreKit: Automatic Feature Generation and Selection
Authors: Gilad Katz, Eui Chul Richard Shin, Dawn Song
Date: 2016-01-01 00:00:00
Proceeding: ICDM 2016 2016 1
Tasks: Automated Feature Engineering, BIG-bench Machine Learning, Classification, feature selection, General Classification

Feature generation is one of the challenging aspects of machine learning. We present ExploreKit, a framework for automated feature generation. ExploreKit generates a large set of candidate features by combining information in the original features, with the aim of maximizing predictive performance according to user-selected criteria. To overcome the exponential growth of the feature space, ExploreKit uses a novel machine learning-based feature selection approach to predict the usefulness of new candidate features. This approach enables efficient identification of the new features and produces superior results compared to existing feature selection solutions. We demonstrate the effectiveness and robustness of our approach by conducting an extensive evaluation on 25 datasets and 3 different classification algorithms. We show that ExploreKit can achieve classification-error reduction of 20% overall.
Deep Feature Synthesis: Towards Automating Data Science Endeavors
Authors: James Max Kanter, Kalyan Veeramachaneni
Date: 2015-01-01 00:00:00
Proceeding: DSAA 2015 2015 1
Tasks: Automated Feature Engineering

In this paper, we develop the Data Science Machine, which is able to derive predictive models from raw data automatically. To achieve this automation, we first propose and develop the Deep Feature Synthesis algorithm for automatically generating features for relational datasets. The algorithm follows relationships in the data to a base field, and then sequentially applies mathematical functions along that path to create the final feature. Second, we implement a generalizable machine learning pipeline and tune it using a novel Gaussian Copula process based approach. We entered the Data Science Machine in 3 data science competitions that featured 906 other data science teams. Our approach beats 615 teams in these data science competitions. In 2 of the 3 competitions we beat a majority of competitors, and in the third, we achieved 94% of the best competitor's score. In the best case, with an ongoing competition, we beat 85.6% of the teams and achieved 95.7% of the top submissions score.
Cognito: Automated Feature Engineering for Supervised Learning
Authors: Udayan Khurana, Deepak Turaga, Horst Samulowitz, Srinivasan Parthasrathy
Date: 2016-01-01 00:00:00
Proceeding: ICDMW 2016 2016 1
Tasks: Automated Feature Engineering, Feature Engineering, Model Selection

Feature engineering involves constructing novel features from given data with the goal of improving predictive learning performance. Feature engineering is predominantly a human-intensive and time consuming step that is central to the data science workflow. In this paper, we present a novel system called "Cognito", that performs automatic feature engineering on a given dataset for supervised learning. The system explores various feature construction choices in a hierarchical and non-exhaustive manner, while progressively maximizing the accuracy of the model through a greedy exploration strategy. Additionally, the system allows users to specify domain or data specific choices to prioritize the exploration. Cognito is capable of handling large datasets through sampling and built-in parallelism, and integrates well with a state-of-the-art model selection strategy. We present the design and operation of Cognito, along with experimental results on eight real datasets to demonstrate its efficacy.
Learning Feature Engineering for Classification
Authors: Fatemeh Nargesian, Horst Samulowitz, Udayan Khurana, Elias B. Khalil, Deepak Turaga
Date: 2017-01-01 00:00:00
Proceeding: IJCAI 2017 2017 1
Tasks: Automated Feature Engineering, Classification, Feature Engineering, feature selection, General Classification

Feature engineering is the task of improving predictive modelling performance on a dataset by transforming its feature space. Existing approaches to automate this process rely on either transformed feature space exploration through evaluation-guided search, or explicit expansion of datasets with all transformed features followed by feature selection. Such approaches incur high computational costs in runtime and/or memory. We present a novel technique, called Learning Feature Engineering (LFE), for automating feature engineering in classification tasks. LFE is based on learning the effectiveness of applying a transformation (e.g., arithmetic or aggregate operators) on numerical features, from past feature engineering experiences. Given a new dataset, LFE recommends a set of useful transformations to be applied on features without relying on model evaluation or explicit feature expansion and selection. Using a collection of datasets, we train a set of neural networks, which aim at predicting the transformation that impacts classification performance positively. Our empirical results show that LFE outperforms other feature engineering approaches for an overwhelming majority (89%) of the datasets from various sources while incurring a substantially lower computational cost.
Feature Selection as a One-Player Game
Authors: Romaric Gaudel, Michèle Sebag
Date: 2010-05-17 00:00:00
Proceeding: International Conference on Machine Learning 2010 2010 5
Tasks: Automated Feature Engineering, feature selection, Reinforcement Learning, Reinforcement Learning (RL)

This paper formalizes Feature Selection as a Reinforcement Learning problem, leading to a provably optimal though intractable selection policy. As a second contribution, this paper presents an approximation thereof, based on a one-player game approach and relying on the Monte-Carlo tree search UCT (Upper Confidence Tree) proposed by Kocsis and Szepesvari (2006). The Feature Uct SElection (FUSE) algorithm extends UCT to deal with i) a finite unknown horizon (the target number of relevant features); ii) the huge branching factor of the search tree, reflecting the size of the feature set. Finally, a frugal reward function is proposed as a rough but unbiased estimate of the relevance of a feature subset. A proof of concept of FUSE is shown on benchmark data sets.
Evolving Neural Networks through Augmenting Topologies
Authors: Kenneth O. Stanley, Risto Miikkulainen
Date: 2006-03-13 00:00:00
Proceeding: Evolutionary Computation 2002 2006 3
Tasks: Neural Architecture Search, Reinforcement Learning

An important question in neuroevolution is how to gain an advantage from evolving neural network topologies along with weights. We present a method, NeuroEvolution of Augmenting Topologies (NEAT), which outperforms the best fixed-topology method on a challenging benchmark reinforcement learning task. We claim that the increased efficiency is due to (1) employing a principled method of crossover of different topologies, (2) protecting structural innovation using speciation, and (3) incrementally growing from minimal structure. We test this claim through a series of ablation studies that demonstrate that each component is necessary to the system as a whole and to each other. What results is significantly faster learning. NEAT is also an important contribution to GAs because it shows how it is possible for evolution to both optimize and complexify solutions simultaneously, offering the possibility of evolving increasingly complex solutions over generations, and strengthening the analogy with biological evolution.
AutoCompete: A Framework for Machine Learning Competitions
Authors: Abhishek Thakur, Artus Krohn-Grimberghe
Date: 2015-07-08 00:00:00
Proceeding: ICML 2015 2015 7
Tasks: AutoML, BIG-bench Machine Learning

In this paper, we propose AutoCompete, a highly automated machine learning framework for tackling machine learning competitions. This framework has been learned by us, validated and improved over a period of more than two years by participating in online machine learning competitions. It aims at minimizing human interference required to build a first useful predictive model and to assess the practical difficulty of a given machine learning challenge. The proposed system helps in identifying data types, choosing a machine learning model, tuning hyper-parameters, avoiding over-fitting and optimization for a provided evaluation metric. We also observe that the proposed system produces better (or comparable) results with less runtime as compared to other approaches.
Scalable Hyperparameter Optimization with Products of Gaussian Process Experts
Authors: Nicolas Schilling, Martin Wistuba, Lars Schmidt-Thieme
Date: 2016-09-04 00:00:00
Proceeding: ECML PKDD 2016 2016 9
Tasks: Gaussian Processes, Hyperparameter Optimization

In machine learning, hyperparameter optimization is a challenging but necessary task that is usually approached in a computationally expensive manner such as grid-search. Out of this reason, surrogate based black-box optimization techniques such as sequential model-based optimization have been proposed which allow for a faster hyperparameter optimization. Recent research proposes to also integrate hyperparameter performances on past data sets to allow for a faster and more efficient hyperparameter optimization.

In this paper, we use products of Gaussian process experts as surrogate models for hyperparameter optimization. Naturally, Gaussian processes are a decent choice as they offer good prediction accuracy as well as estimations about their uncertainty. Additionally, their hyperparameters can be tuned very effectively. However, in the light of large meta data sets, learning a single Gaussian process is not feasible as it involves inversion of a large kernel matrix. This directly limits their usefulness for hyperparameter optimization if large scale hyperparameter performances on past data sets are given.

By using products of Gaussian process experts the scalability issues can be circumvented, however, this usually comes with the price of having less predictive accuracy. In our experiments, we show empirically that products of experts nevertheless perform very well compared to a variety of published surrogate models. Thus, we propose a surrogate model that performs as well as the current state of the art, is scalable to large scale meta knowledge, does not include hyperparameters itself and finally is even very easy to parallelize.
Automatic Frankensteining: Creating Complex Ensembles Autonomously
Authors: Martin Wistuba, Nicolas Schilling, Lars Schmidt-Thieme
Date: 2017-01-01 00:00:00
Proceeding: SIAM 2017 2017 1
Tasks: AutoML, BIG-bench Machine Learning, CPU

Automating machine learning by providing techniques that autonomously find the best algorithm, hyperparameter configuration and preprocessing is helpful for both researchers and practitioners. Therefore, it is not surprising that automated machine learning has become a very interesting field of research. While current research is mainly focusing on finding good pairs of algorithms and hyperparameter configurations, we will present an approach that automates the process of creating a top performing ensemble of several layers, different algorithms and hyperparameter configurations. These kinds of ensembles are called jokingly Frankenstein ensembles and proved their benefit on versatile data sets in many machine learning challenges. We compare our approach Automatic Frankensteining with the current state of the art for automated machine learning on 80 different data sets and can show that it outperforms them on the majority using the same training time. Furthermore, we compare Automatic Frankensteining on a large scale data set to more than 3,500 machine learning expert teams and are able to outperform more than 3,000 of them within 12 CPU hours.
Feature Selection with the Boruta Package
Authors: Miron B. Kursa, Witold R. Rudnicki
Date: 2010-09-01 00:00:00
Proceeding: Journal of Statistical Software 2010 2010 9
Tasks: feature selection, General Classification

This article describes a R package Boruta, implementing a novel feature selection algorithm for finding all relevant variables. The algorithm is designed as a wrapper around a Random Forest classification algorithm. It iteratively removes the features which are proved by a statistical test to be less relevant than random probes. The Boruta package provides a convenient interface to the algorithm. The short description of the algorithm and examples of its application are presented.
Mutual Information between Discrete and Continuous Data Sets
Authors: Brian C. Ross
Date: 2014-02-14 00:00:00
Proceeding: 2014 2014 2

Mutual information (MI) is a powerful method for detecting relationships between data sets. There are accurate methods for estimating MI that avoid problems with “binning” when both data sets are discrete or when both data sets are continuous. We present an accurate, non-binning MI estimator for the case of one discrete data set and one continuous data set. This case applies when measuring, for example, the relationship between base sequence and gene expression level, or the effect of a cancer drug on patient survival time. We also show how our method can be adapted to calculate the Jensen–Shannon divergence of two or more data sets.
An Information Theory-Based Feature Selection Framework for Big Data Under Apache Spark
Authors: Sergio Ramírez-Gallego, Héctor Mouriño-Talín, David Martínez-Rego, Verónica Bolón-Canedo, José Manuel Benítez, Amparo Alonso-Betanzos, Francisco Herrera
Date: 2017-07-06 00:00:00
Proceeding: IEEE 2017 2017 7
Tasks: Dimensionality Reduction, feature selection

With the advent of extremely high dimensional datasets, dimensionality reduction techniques are becoming mandatory. Of the many techniques available, feature selection (FS) is of growing interest for its ability to identify both relevant features and frequently repeated instances in huge datasets. We aim to demonstrate that standard FS methods can be parallelized in big data platforms like Apache Spark so as to boost both performance and accuracy. We propose a distributed implementation of a generic FS framework that includes a broad group of well-known information theory-based methods. Experimental results for a broad set of real-world datasets show that our distributed framework is capable of rapidly dealing with ultrahigh-dimensional datasets as well as those with a huge number of samples, outperforming the sequential version in all the cases studied.
Automatic Differentiation in PyTorch
Authors: Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer
Date: 2017-10-28 00:00:00
Proceeding: NIPS 2017 2017 10
Tasks: Clustering, CPU, Dimensionality Reduction, General Classification, GPU

In this article, we describe an automatic differentiation module of PyTorch — a library designed to enable rapid research on machine learning models. It builds upon a few projects, most notably Lua Torch, Chainer, and HIPS Autograd, and provides a high performance environment with easy access to automatic differentiation of models executed on different devices (CPU and GPU). To make prototyping easier, PyTorch does not follow the symbolic approach used in many other deep learning frameworks, but focuses on differentiation of purely imperative programs, with a focus on extensibility and low overhead. Note that this preprint is a draft of certain sections from an upcoming paper covering all PyTorch features.
CNTK: Microsoft's Open-Source Deep-Learning Toolkit
Authors: Frank Seide, Amit Agarwal
Date: 2016-08-01 00:00:00
Proceeding: ACM SIGKDD 2016 8
Tasks: Clustering, Deep Learning, Dimensionality Reduction, General Classification, GPU, speech-recognition

This tutorial will introduce the Computational Network Toolkit, or CNTK, Microsoft's cutting-edge open-source deep-learning toolkit for Windows and Linux. CNTK is a powerful computation-graph based deep-learning toolkit for training and evaluating deep neural networks. Microsoft product groups use CNTK, for example to create the Cortana speech models and web ranking. CNTK supports feed-forward, convolutional, and recurrent networks for speech, image, and text workloads, also in combination. Popular network types are supported either natively (convolution) or can be described as a CNTK configuration (LSTM, sequence-to-sequence). CNTK scales to multiple GPU servers and is designed around efficiency. The tutorial will give an overview of CNTK's general architecture and describe the specific methods and algorithms used for automatic differentiation, recurrent-loop inference and execution, memory sharing, on-the-fly randomization of large corpora, and multi-server parallelization. We will then show how typical uses looks like for relevant tasks like image recognition, sequence-to-sequence modeling, and speech recognition.
Chainer: a Next-Generation Open Source Framework for Deep Learning
Authors: Seiya Tokui, Kenta Oono, Shohei Hido, Justin Clayton
Date: 2015-01-01 00:00:00
Proceeding: NIPS 2015 1
Tasks: Clustering, Deep Learning, Dimensionality Reduction, General Classification

Software frameworks for neural networks play key roles in the development and application of deep learning methods. However, as new types of deep learning models are developed, existing frameworks designed for convolutional neural networks are becoming less useful. In this paper, we introduce Chainer, a Python based, standalone open source framework for deep learning models. Chainer provides a flexible, intuitive, and high performance means of implementing a full range of deep learning models, including state-of-the-art models such as recurrent neural networks and variational autoencoders.
Fully Decoupled Neural Network Learning Using Delayed Gradients
Authors: Huiping Zhuang, Yi Wang, Qinglai Liu, Shuai Zhang, Zhiping Lin
Date: 2019-06-21 00:00:00

Training neural networks with back-propagation (BP) requires a sequential passing of activations and gradients, which forces the network modules to work in a synchronous fashion. This has been recognized as the lockings (i.e., the forward, backward and update lockings) inherited from the BP. In this paper, we propose a fully decoupled training scheme using delayed gradients (FDG) to break all these lockings. The FDG splits a neural network into multiple modules and trains them independently and asynchronously using different workers (e.g., GPUs). We also introduce a gradient shrinking process to reduce the stale gradient effect caused by the delayed gradients. In addition, we prove that the proposed FDG algorithm guarantees a statistical convergence during training. Experiments are conducted by training deep convolutional neural networks to perform classification tasks on benchmark datasets, showing comparable or better results against the state-of-the-art methods as well as the BP in terms of both generalization and acceleration abilities. In particular, we show that the FDG is also able to train very wide networks (e.g., WRN-28-10) and extremely deep networks (e.g., ResNet-1202). Code is available at https://github.com/ZHUANGHP/FDG.
Replay attack detection with complementary high-resolution information using end-to-end DNN for the ASVspoof 2019 Challenge
Authors: Jee-weon Jung, Hye-jin Shim, Hee-Soo Heo, Ha-Jin Yu
Date: 2019-04-23 00:00:00

In this study, we concentrate on replacing the process of extracting hand-crafted acoustic feature with end-to-end DNN using complementary high-resolution spectrograms. As a result of advance in audio devices, typical characteristics of a replayed speech based on conventional knowledge alter or diminish in unknown replay configurations. Thus, it has become increasingly difficult to detect spoofed speech with a conventional knowledge-based approach. To detect unrevealed characteristics that reside in a replayed speech, we directly input spectrograms into an end-to-end DNN without knowledge-based intervention. Explorations dealt in this study that differentiates from existing spectrogram-based systems are twofold: complementary information and high-resolution. Spectrograms with different information are explored, and it is shown that additional information such as the phase information can be complementary. High-resolution spectrograms are employed with the assumption that the difference between a bona-fide and a replayed speech exists in the details. Additionally, to verify whether other features are complementary to spectrograms, we also examine raw waveform and an i-vector based system. Experiments conducted on the ASVspoof 2019 physical access challenge show promising results, where t-DCF and equal error rates are 0.0570 and 2.45 % for the evaluation set, respectively.