pwc-archive/papers-with-abstracts huggingface dataset:
Towards Explaining Hyperparameter Optimization via Partial Dependence Plots
Authors: Julia Moosbauer, Julia Herbinger, Giuseppe Casalicchio, Marius Lindauer, Bernd Bischl
Date: 2021-05-20 00:00:00
Proceeding: ICML Workshop AutoML 2021 7
Tasks: Bayesian Optimization, BIG-bench Machine Learning, Hyperparameter Optimization, Interpretable Machine Learning

Automated hyperparameter optimization (HPO) can support practitioners to obtain peak performance in machine learning models.
However, there is often a lack of valuable insights into the effects of different hyperparameters on the final model performance.
This lack of comprehensibility and transparency makes it difficult to trust and understand the automated HPO process and its results.
We suggest using interpretable machine learning (IML) to gain insights from the experimental data obtained during HPO and especially discuss the popular case of Bayesian optimization (BO).
BO tends to focus on promising regions with potential high-performance configurations and thus induces a sampling bias.
Hence, many IML techniques, like Partial Dependence Plots (PDP), carry the risk of generating biased interpretations.
By leveraging the posterior uncertainty of the BO surrogate model, we introduce a variant of the PDP with estimated confidence bands.
In addition, we propose to partition the hyperparameter space to obtain more confident and reliable PDPs in relevant sub-regions.
In an experimental study, we provide quantitative evidence for the increased quality of the PDPs within sub-regions.
Leveraging Theoretical Tradeoffs in Hyperparameter Selection for Improved Empirical Performance
Authors: Parikshit Ram, Alexander G. Gray, Horst Samulowitz
Date: 2021-05-20 00:00:00
Proceeding: ICML Workshop AutoML 2021 7
Tasks: Hyperparameter Optimization

The tradeoffs in the excess risk incurred from data-driven learning of a single model has been studied by decomposing the excess risk into approximation, estimation and optimization errors. In this paper, we focus on the excess risk incurred in data-driven hyperparameter optimization (HPO) and its interaction with approximate empirical risk minimization (ERM) necessitated by large data. We present novel bounds for the excess risk in various common scenarios in HPO. Based on these results, we propose practical heuristics that allow us to improve performance or reduce computational overhead of data-driven HPO, demonstrating over $2 \times$ speedup with no loss in predictive performance in our preliminary results.
AutoML Adoption in ML Software
Authors: Koen van der Blom, Alex Serban, Holger Hoos, Joost Visser
Date: 2021-05-20 00:00:00
Proceeding: ICML Workshop AutoML 2021 7
Tasks: AutoML

Machine learning (ML) has become essential to a vast range of applications, while ML experts are in short supply. To alleviate this problem, AutoML aims to make ML easier and more efficient to use. Even so, it is not clear to which extent AutoML techniques are actually adopted in an engineering context, nor what facilitates or inhibits adoption. To study this, we define AutoML engineering practices, measure their adoption through surveys, and distil first insights into factors influencing adoption from two initial interviews. Depending on the practice, results show around 20 to 30% of the respondents have not adopted it at all and many more only partially, leaving substantial room for increases in adoption. The interviews indicate adoption may in part be inhibited by usability issues with AutoML frameworks and the increased computational resources needed for adoption.
Multimodal AutoML on Structured Tables with Text Fields
Authors: Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, Alex Smola
Date: 2021-05-20 00:00:00
Proceeding: ICML Workshop AutoML 2021 7
Tasks: AutoML

We  design  automated  supervised  learning  systems  for  data  tables  that  not  only  contain numeric/categorical columns,  but  text  fields  as  well.  Here  we  assemble  15  multimodal data tables that each contain some text fields and stem from a real business application. Over this benchmark, we evaluate numerous multimodal AutoML strategies, including a standard two-stage approach where NLP is used to featurize the text such that AutoML for tabular data can then be applied. We propose various practically superior strategies based on multimodal adaptations of Transformer networks and stack ensembling of these networks with classical tabular models.  Beyond performing the best in our benchmark, our proposed (fully automated) methodology manages to rank 1st place (against human data scientists) when fit to the raw tabular/text data in two MachineHack prediction competitions  and 2nd place (out of 2380 teams) in Kaggle’s Mercari Price Suggestion Challenge.
Dynamic Pruning of a Neural Network via Gradient Signal-to-Noise Ratio
Authors: Julien Niklas Siems, Aaron Klein, Cedric Archambeau, Maren Mahsereci
Date: 2021-05-20 00:00:00
Proceeding: ICML Workshop AutoML 2021 7

While training highly overparameterized neural networks is common practice in deep learning, research into post-hoc weight-pruning suggests that more than 90% of parameters can be removed without loss in predictive performance.  To save resources, zero-shot and one-shot pruning attempt to find such a sparse representation at initialization or at an early stage of training. Though efficient, there is no justification, why the sparsity structure should not change during training. Dynamic sparsity pruning undoes this limitation and allows to adapt the structure of the sparse neural network during training. Recent approaches rely on weight magnitude pruning, which has been shown to be sub-optimal when applied at earlier training stages. In this work we propose to use the gradient noise to make pruning decisions. The procedure enables us to automatically adjust the sparsity during training without imposing a hand-designed sparsity schedule, while at the same time being able to recover from previous pruning decisions by unpruning connections as necessary. We evaluate our new method on image and tabular datasets and demonstrate that we reach similar performance as the dense model from which extract the sparse network, while exposing less hyperparameters than other dynamic sparsity methods.
Towards Model Selection using Learning Curve Cross-Validation
Authors: Felix Mohr, Jan N. van Rijn
Date: 2021-05-20 00:00:00
Proceeding: ICML Workshop AutoML 2021 7
Tasks: AutoML, model, Model Selection

Cross-validation (CV) methods such as leave-one-out cross-validation, k-fold cross-validation, and Monte-Carlo cross-validation estimate the predictive performance of a learner by repeatedly training it on a large portion of the given data and testing on the remaining data. These techniques have two drawbacks. First, they can be unnecessarily slow on large datasets. Second, providing only point estimates, they give almost no insights into the learning process of the validated algorithm. In this paper, we propose a new approach for validation based on learning curves (LCCV). Instead of creating train-test splits with a large portion of training data, LCCV iteratively increases the number of training examples used for training. In the context of model selection, it eliminates models that can be safely dismissed from the candidate pool. We run a large scale experiment on the 67 datasets from the AutoML benchmark, and empirically show that LCCV in over 90\% of the cases leads to similar performance (at most 0.5\% difference) as 10-fold CV, but provides additional insights on the behaviour of a given model. On top of this, LCCV results in runtime reductions between 20% and over 50% on half of the 67 datasets from the AutoML benchmark. This can be incorporated in various AutoML frameworks, to speed up the internal evaluation of candidate models. As such, these results can be used orthogonal to other advances in the field of AutoML.
Bandit Limited Discrepancy Search and Application to Machine Learning Pipeline Optimization
Authors: Akihiro Kishimoto, Djallel Bouneffouf, Radu Marinescu, Parikshit Ram, Ambrish Rawat, Martin Wistuba, Paulito Pedregosa Palmes, Adi Botea
Date: 2021-05-20 00:00:00
Proceeding: ICML Workshop AutoML 2021 7
Tasks: BIG-bench Machine Learning

Optimizing a machine learning (ML) pipeline has been an important topic of AI and ML.  Despite recent progress, this topic remains a challenging problem, due to potentially many combinations to consider as well as slow training and validation.  We present the BLDS algorithm for optimized algorithm selection (ML operations) in a fixed ML pipeline structure.  BLDS performs multi-fidelity optimization for selecting ML algorithms trained with smaller computational overhead, while controlling its pipeline search based on multi-armed bandit and limited discrepancy search.  Our experiments on well-known benchmarks show that BLDS is superior to competing algorithms.
GPy-ABCD: A Configurable Automatic Bayesian Covariance Discovery Implementation
Authors: Thomas Fletcher, Alan Bundy, Kwabena Nuamah
Date: 2021-05-20 00:00:00
Proceeding: ICML Workshop AutoML 2021 7
Tasks: Gaussian Processes

Gaussian Processes (GPs) are a very flexible class of nonparametric models frequently used in supervised learning tasks because of their ability to fit data with very few assumptions, namely just the type of correlation (kernel) the data is expected to display. Automatic Bayesian Covariance Discovery (ABCD) is an iterative GP regression framework aimed at removing the requirement for even this initial correlation form assumption. An original ABCD implementation exists and is a complex stand-alone system designed to produce long-form text analyses of provided data. This paper presents a lighter, more functional and configurable implementation of the ABCD idea, outputting only fit models and short descriptions: the Python package GPy-ABCD, which was developed as part of an adaptive modelling component for the FRANK query-answering system. It uses a revised model-space search algorithm and removes a search bias which was required in order to retain model explainability in the original system.
Discovering Weight Initializers with Meta Learning
Authors: Dmitry Baranchuk, Artem Babenko
Date: 2021-05-20 00:00:00
Proceeding: ICML Workshop AutoML 2021 7
Tasks: Meta-Learning

Deep neural network training largely depends on the choice of initial weight distribution. However, this choice can often be nontrivial.  Existing theoretical results for this problem mostly cover simple architectures, e.g., feedforward networks with ReLU activations.  The architectures  used  for  practical  problems  are  more  complex  and  often  incorporate  many overlapping modules, making them challenging for theoretical analysis.  Therefore, practitioners have to use heuristic initializers with questionable optimality and stability.  In this study, we propose a task-agnostic approach that discovers initializers for specific network architectures and optimizers by learning the initial weight distributions directly through the use of Meta-Learning.  In several supervised and unsupervised learning scenarios, we show the advantage of our initializers in terms of both faster convergence and higher model performance.
Agent Forecasting at Flexible Horizons using ODE Flows
Authors: Alexander Radovic, JiaWei He, Janahan Ramanan, Marcus A Brubaker, Andreas Lehrmann
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7
Tasks: Position, Prediction

In this work we describe OMEN, a neural ODE based normalizing flow for the prediction of marginal distributions at flexible evaluation horizons, and apply it to agent position forecasting.
 OMEN's architecture embeds an assumption that marginal distributions of a given agent moving forward in time are related, allowing for an efficient representation of marginal distributions through time and allowing for reliable interpolation between prediction horizons seen in training.
 Experiments on a popular agent forecasting dataset demonstrate significant improvements over most baseline approaches, and comparable performance to the state of the art while providing the new functionality of reliable interpolation of predicted marginal distributions between prediction horizons as demonstrated with synthetic data.
Why be adversarial? Let's cooperate!: Cooperative Dataset Alignment via JSD Upper Bound
Authors: Wonwoong Cho, Ziyu Gong, David I. Inouye
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7
Tasks: Domain Adaptation, Unsupervised Domain Adaptation

Unsupervised dataset alignment estimates a transformation that maps two or more source domains to a shared aligned domain given only the domain datasets. This task has many applications including generative modeling, unsupervised domain adaptation, and socially aware learning. Most prior works use adversarial learning (i.e., min-max optimization), which can be challenging to optimize and evaluate. A few recent works explore non-adversarial flow-based (i.e., invertible) approaches, but they lack a unified perspective. Therefore, we propose to unify and generalize previous flow-based approaches under a single non-adversarial framework, which we prove is equivalent to minimizing an upper bound on the Jensen-Shannon Divergence (JSD). Importantly, our problem reduces to a min-min, i.e., cooperative, problem and can provide a natural evaluation metric for unsupervised dataset alignment.
RAD-TTS: Parallel Flow-Based TTS with Robust Alignment Learning and Diverse Synthesis
Authors: Kevin J. Shih, Rafael Valle, Rohan Badlani, Adrian Lancucki, Wei Ping, Bryan Catanzaro
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7
Tasks: Diversity, Rhythm, Speech Synthesis, Text-To-Speech Synthesis

This work introduces a predominantly parallel, end-to-end TTS model based on normalizing flows.
 It extends prior parallel approaches by additionally modeling speech rhythm as a separate generative distribution to facilitate variable token duration during inference. We further propose a robust framework for the on-line extraction of speech-text alignments -- a critical yet highly unstable learning problem in end-to-end TTS frameworks. Our experiments demonstrate that our proposed techniques yield improved alignment quality, better output diversity compared to controlled baselines.
Manifold Density Estimation via Generalized Dequantization
Authors: James Brofos, Marcus A Brubaker, Roy R Lederman
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7
Tasks: Density Estimation

Density estimation is an important technique for characterizing distributions given observations. Much existing research on density estimation has focused on cases wherein the data lies in a Euclidean space. However, some kinds of data are not well-modeled by supposing that their underlying geometry is Euclidean. Instead, it can be useful to model such data as lying on a {\it manifold} with some known structure. For instance, some kinds of data may be known to lie on the surface of a sphere. We study the problem of estimating densities on manifolds. We propose a method, inspired by the literature on "dequantization," which we interpret through the lens of a coordinate transformation of an ambient Euclidean space and a smooth manifold of interest. Using methods from normalizing flows, we apply this method to the dequantization of smooth manifold structures in order to model densities on the sphere, tori, and the orthogonal group.
$\alpha$-VAEs : Optimising variational inference by learning data-dependent divergence skew
Authors: Jacob Deasy, Tom Andrew McIver, Nikola Simidjievski, Pietro Lio
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7
Tasks: Denoising, Variational Inference

 The {\em skew-geometric Jensen-Shannon divergence} $\left(\textrm{JS}^{\textrm{G}_{\alpha}}\right)$ allows for an intuitive interpolation between forward and reverse Kullback-Leibler (KL) divergence based on the skew parameter $\alpha$. While the benefits of the skew in $\textrm{JS}^{\textrm{G}_{\alpha}}$ are clear---balancing forward/reverse KL in a comprehensible manner---the choice of optimal skew remains opaque and requires an expensive grid search. In this paper we introduce $\alpha$-VAEs, which extend the $\textrm{JS}^{\textrm{G}_{\alpha}}$ variational autoencoder by allowing for learnable, and therefore data-dependent, skew. We motivate the use of a parameterised skew in the dual divergence by analysing trends dependent on data complexity in synthetic examples. We also prove and discuss the dependency of the divergence minimum on the input data and encoder parameters, before empirically demonstrating that this dependency does not reduce to either direction of KL divergence for benchmark datasets. Finally, we demonstrate that optimised skew values consistently converge across a range of initial values and provide improved denoising and reconstruction properties. These render $\alpha$-VAEs an efficient and practical modelling choice across a range of tasks, datasets, and domains.
Conformal Embedding Flows: Tractable Density Estimation on Learned Manifolds
Authors: Brendan Leigh Ross, Jesse C Cresswell
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7
Tasks: Density Estimation

Normalizing flows are generative models that provide tractable density estimation by transforming a simple distribution into a complex one. However, flows cannot directly model data supported on an unknown low-dimensional manifold. We propose Conformal Embedding Flows, which learn low-dimensional manifolds with tractable densities. We argue that composing a standard flow with a trainable conformal embedding is the most natural way to model manifold-supported data. To this end, we present a series of conformal building blocks and demonstrate experimentally that flows can model manifold-supported distributions without sacrificing tractable likelihoods.
General Invertible Transformations for Flow-based Generative Modeling
Authors: Jakub Mikolaj Tomczak
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7

In this paper, we present a new class of invertible transformations with an application to flow-based generative models. We indicate that many well-known invertible transformations in reversible logic and reversible neural networks could be derived from our proposition. Next, we propose two new coupling layers that are important building blocks of flow-based generative models. In the experiments on digit data, we present how these new coupling layers could be used in Integer Discrete Flows (IDF), and that they achieve better results than standard coupling layers used in IDF and RealNVP.
Representation Learning in Continuous-Time Score-Based Generative Models
Authors: Korbinian Abstreiter, Stefan Bauer, Arash Mehrjou
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7
Tasks: Denoising, Representation Learning

Score-based methods represented as stochastic differential equations on a continuous time domain have recently proven successful as a non-adversarial generative model.
Training such models relies on denoising score matching, which can be seen as multi-scale denoising autoencoders.
Here, we augment the denoising score-matching framework to enable representation learning without any supervised signal.
GANs and VAEs learn representations by directly transforming latent codes to data samples.
In contrast, score-based representation learning relies on a new formulation of the denoising score-matching objective and thus encodes information needed for denoising.
We show how this difference allows for manual control of the level of detail encoded in the representation.
Challenges for BBVI with Normalizing Flows
Authors: Akash Kumar Dhaka, Alejandro Catalina, Manushi Welandawe, Michael Riis Andersen, Jonathan H. Huggins, Aki Vehtari
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7
Tasks: Variational Inference

Current black-box variational inference (BBVI) methods require the user to make numerous design choices---such as the selection of variational objective and approximating family---yet there is little principled guidance on how to do so. We develop a conceptual framework and set of experimental tools to understand the effects of these choices, which we leverage to propose best practices for maximizing posterior approximation accuracy. Our approach is based on studying the pre-asymptotic tail behavior of the density ratios between the joint distribution and the variational approximation, then exploiting insights and tools from the importance sampling literature. We focus on normalizing flow models and give recommendations on how to be used(and diagnostics) in BBVI, though we are not limited to them.
Deep Signature Statistics for Likelihood-free Time-series Models
Authors: Joel Dyer, Patrick W Cannon, Sebastian M Schmon
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7
Tasks: Time Series, Time Series Analysis

Simulation-based inference (SBI) has emerged as a family of methods for performing inference on complex simulation models with intractable likelihood functions. A common bottleneck in SBI is the construction of low-dimensional summary statistics of the data. In this respect, time-series data, often being high-dimensional, multivariate, and complex in structure, present a particular challenge. To address this we introduce deep signature statistics, a principled and automated method for combining summary statistic selection for time-series data with neural SBI methods. Our approach leverages deep signature transforms, trained concurrently with a neural density estimator, to produce informative statistics for multivariate sequential data that encode important geometric properties of the underlying path. We obtain competitive results across benchmark models.
Diffeomorphic Explanations with Normalizing Flows
Authors: Ann-Kathrin Dombrowski, Jan E Gerken, Pan Kessel
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7

Normalizing flows are diffeomorphisms which are parameterized by neural networks. As a result, they can induce coordinate transformations in the tangent space of the data manifold. In this work, we demonstrate that such transformations can be used to generate interpretable explanations for decisions of neural networks. More specifically, we perform gradient ascent in the base space of the flow to generate counterfactuals which are classified with great confidence as a specified target class. We analyze this generation process theoretically using Riemannian differential geometry and establish a rigorous theoretical connection between gradient ascent on the data manifold and in the base space of the flow.
Improving Continuous Normalizing Flows using a Multi-Resolution Framework
Authors: Vikram Voleti, Chris Finlay, Adam M Oberman, Christopher Pal
Date: 2021-06-02 00:00:00
Proceeding: ICML Workshop INNF 2021 7
Tasks: Density Estimation, GPU

Recent work has shown that Continuous Normalizing Flows (CNFs) can serve as generative models of images with exact likelihood calculation and invertible generation/density estimation. In this work we introduce a Multi-Resolution variant of such models (MRCNF). We introduce a transformation between resolutions that allows for no change in the log likelihood. We show that this approach yields comparable likelihood values for various image datasets, with improved performance at higher resolutions, with fewer parameters, using only 1 GPU.
Multi-Scale Continuous Normalizing Flows
Authors: Anonymous
Date: 2021-06-02 00:00:00

We introduce a multi-scale variant of Continuous Normalizing Flows, and explore the computation of likelihood values.
We also introduce a Wavelet-version of the model. However, we find that this formulation is flawed in the computation of BPD, and explore ways to alleviate this problem.