pwc-archive/papers-with-abstracts huggingface dataset:
Learning Solution Manifolds for Control Problems via Energy Minimization
Authors: Miguel Zamora, Roi Poranne, Stelian Coros
Date: 2022-03-07 00:00:00
Tasks: Imitation Learning, Model Predictive Control

A variety of control tasks such as inverse kinematics (IK), trajectory optimization (TO), and model predictive control (MPC) are commonly formulated as energy minimization problems. Numerical solutions to such problems are well-established. However, these are often too slow to be used directly in real-time applications. The alternative is to learn solution manifolds for control problems in an offline stage. Although this distillation process can be trivially formulated as a behavioral cloning (BC) problem in an imitation learning setting, our experiments highlight a number of significant shortcomings arising due to incompatible local minima, interpolation artifacts, and insufficient coverage of the state space. In this paper, we propose an alternative to BC that is efficient and numerically robust. We formulate the learning of solution manifolds as a minimization of the energy terms of a control objective integrated over the space of problems of interest. We minimize this energy integral with a novel method that combines Monte Carlo-inspired adaptive sampling strategies with the derivatives used to solve individual instances of the control task. We evaluate the performance of our formulation on a series of robotic control problems of increasing complexity, and we highlight its benefits through comparisons against traditional methods such as behavioral cloning and Dataset aggregation (Dagger).
Multivariate Time Series Forecasting with Latent Graph Inference
Authors: Victor Garcia Satorras, Syama Sundar Rangapuram, Tim Januschowski
Date: 2022-03-07 00:00:00
Tasks: Computational Efficiency, Multivariate Time Series Forecasting, Time Series, Time Series Analysis, Time Series Forecasting

This paper introduces a new approach for Multivariate Time Series forecasting that jointly infers and leverages relations among time series. Its modularity allows it to be integrated with current univariate methods. Our approach allows to trade-off accuracy and computational efficiency gradually via offering on one extreme inference of a potentially fully-connected graph or on another extreme a bipartite graph. In the potentially fully-connected case we consider all pair-wise interactions among time-series which yields the best forecasting accuracy. Conversely, the bipartite case leverages the dependency structure by inter-communicating the N time series through a small set of K auxiliary nodes that we introduce. This reduces the time and memory complexity w.r.t. previous graph inference methods from O(N^2) to O(NK) with a small trade-off in accuracy. We demonstrate the effectiveness of our model in a variety of datasets where both of its variants perform better or very competitively to previous graph inference methods in terms of forecasting accuracy and time efficiency.
Estimation under Model Misspecification with Fake Features
Authors: Martin Hellkvist, Ayça Özçelikkale, Anders Ahlén
Date: 2022-03-07 00:00:00
Tasks: model

We consider estimation under model misspecification where there is a model mismatch between the underlying system, which generates the data, and the model used during estimation. We propose a model misspecification framework which enables a joint treatment of the model misspecification types of having fake features as well as incorrect covariance assumptions on the unknowns and the noise. We present a decomposition of the output error into components that relate to different subsets of the model parameters corresponding to underlying, fake and missing features. Here, fake features are features which are included in the model but are not present in the underlying system. Under this framework, we characterize the estimation performance and reveal trade-offs between the number of samples, number of fake features, and the possibly incorrect noise level assumption. In contrast to existing work focusing on incorrect covariance assumptions or missing features, fake features is a central component of our framework. Our results show that fake features can significantly improve the estimation performance, even though they are not correlated with the features in the underlying system. In particular, we show that the estimation error can be decreased by including more fake features in the model, even to the point where the model is overparametrized, i.e., the model contains more unknowns than observations.
Discovering Inductive Bias with Gibbs Priors: A Diagnostic Tool for Approximate Bayesian Inference
Authors: Luca Rendsburg, Agustinus Kristiadi, Philipp Hennig, Ulrike Von Luxburg
Date: 2022-03-07 00:00:00
Tasks: Bayesian Inference, Diagnostic, Inductive Bias

Full Bayesian posteriors are rarely analytically tractable, which is why real-world Bayesian inference heavily relies on approximate techniques. Approximations generally differ from the true posterior and require diagnostic tools to assess whether the inference can still be trusted. We investigate a new approach to diagnosing approximate inference: the approximation mismatch is attributed to a change in the inductive bias by treating the approximations as exact and reverse-engineering the corresponding prior. We show that the problem is more complicated than it appears to be at first glance, because the solution generally depends on the observation. By reframing the problem in terms of incompatible conditional distributions we arrive at a natural solution: the Gibbs prior. The resulting diagnostic is based on pseudo-Gibbs sampling, which is widely applicable and easy to implement. We illustrate how the Gibbs prior can be used to discover the inductive bias in a controlled Gaussian setting and for a variety of Bayesian models and approximations.
High-Resolution Peak Demand Estimation Using Generalized Additive Models and Deep Neural Networks
Authors: Jonathan Berrisch, Michał Narajewski, Florian Ziel
Date: 2022-03-07 00:00:00
Tasks: Additive models, Load Forecasting

This paper covers predicting high-resolution electricity peak demand features given lower-resolution data. This is a relevant setup as it answers whether limited higher-resolution monitoring helps to estimate future high-resolution peak loads when the high-resolution data is no longer available. That question is particularly interesting for network operators considering replacing high-resolution monitoring predictive models due to economic considerations. We propose models to predict half-hourly minima and maxima of high-resolution (every minute) electricity load data while model inputs are of a lower resolution (30 minutes). We combine predictions of generalized additive models (GAM) and deep artificial neural networks (DNN), which are popular in load forecasting. We extensively analyze the prediction models, including the input parameters' importance, focusing on load, weather, and seasonal effects. The proposed method won a data competition organized by Western Power Distribution, a British distribution network operator. In addition, we provide a rigorous evaluation study that goes beyond the competition frame to analyze the models' robustness. The results show that the proposed methods are superior to the competition benchmark concerning the out-of-sample root mean squared error (RMSE). This holds regarding the competition month and the supplementary evaluation study, which covers an additional eleven months. Overall, our proposed model combination reduces the out-of-sample RMSE by 57.4\% compared to the benchmark.
Neural Enhancement of Factor Graph-based Symbol Detection
Authors: Luca Schmid, Laurent Schmalen
Date: 2022-03-07 00:00:00

We study the application of the factor graph framework for symbol detection on linear inter-symbol interference channels. Cyclic factor graphs have the potential to yield low-complexity symbol detectors, but are suboptimal if the ubiquitous sum-product algorithm is applied. In this paper, we present and evaluate strategies to improve the performance of cyclic factor graph-based symbol detection algorithms by means of neural enhancement. In particular, we apply neural belief propagation as an effective way to counteract the effect of cycles within the factor graph. We further propose the application and optimization of a linear preprocessor of the channel output. By modifying the observation model, the preprocessing can effectively change the underlying factor graph, thereby significantly improving the detection performance as well as reducing the complexity.
Automated Few-Shot Time Series Forecasting based on Bi-level Programming
Authors: Jiangjiao Xu, Ke Li
Date: 2022-03-07 00:00:00
Tasks: Decision Making, Few-Shot Learning, Hyperparameter Optimization, Meta-Learning, Time Series, Time Series Analysis, Time Series Forecasting

New micro-grid design with renewable energy sources and battery storage systems can help improve greenhouse gas emissions and reduce the operational cost. To provide an effective short-/long-term forecasting of both energy generation and load demand, time series predictive modeling has been one of the key tools to guide the optimal decision-making for planning and operation. One of the critical challenges of time series renewable energy forecasting is the lack of historical data to train an adequate predictive model. Moreover, the performance of a machine learning model is sensitive to the choice of its corresponding hyperparameters. Bearing these considerations in mind, this paper develops a BiLO-Auto-TSF/ML framework that automates the optimal design of a few-shot learning pipeline from a bi-level programming perspective. Specifically, the lower-level meta-learning helps boost the base-learner to mitigate the small data challenge while the hyperparameter optimization at the upper level proactively searches for the optimal hyperparameter configurations for both base- and meta-learners. Note that the proposed framework is so general that any off-the-shelf machine learning method can be used in a plug-in manner. Comprehensive experiments fully demonstrate the effectiveness of our proposed BiLO-Auto-TSF/ML framework to search for a high-performance few-shot learning pipeline for various energy sources.
Dynamic ConvNets on Tiny Devices via Nested Sparsity
Authors: Matteo Grimaldi, Luca Mocerino, Antonio Cipolletta, Andrea Calimera
Date: 2022-03-07 00:00:00
Tasks: image-classification, Image Classification, object-detection, Object Detection

This work introduces a new training and compression pipeline to build Nested Sparse ConvNets, a class of dynamic Convolutional Neural Networks (ConvNets) suited for inference tasks deployed on resource-constrained devices at the edge of the Internet-of-Things. A Nested Sparse ConvNet consists of a single ConvNet architecture containing N sparse sub-networks with nested weights subsets, like a Matryoshka doll, and can trade accuracy for latency at run time, using the model sparsity as a dynamic knob. To attain high accuracy at training time, we propose a gradient masking technique that optimally routes the learning signals across the nested weights subsets. To minimize the storage footprint and efficiently process the obtained models at inference time, we introduce a new sparse matrix compression format with dedicated compute kernels that fruitfully exploit the characteristic of the nested weights subsets. Tested on image classification and object detection tasks on an off-the-shelf ARM-M7 Micro Controller Unit (MCU), Nested Sparse ConvNets outperform variable-latency solutions naively built assembling single sparse models trained as stand-alone instances, achieving (i) comparable accuracy, (ii) remarkable storage savings, and (iii) high performance. Moreover, when compared to state-of-the-art dynamic strategies, like dynamic pruning and layer width scaling, Nested Sparse ConvNets turn out to be Pareto optimal in the accuracy vs. latency space.
Regularising for invariance to data augmentation improves supervised learning
Authors: Aleksander Botev, Matthias Bauer, Soham De
Date: 2022-03-07 00:00:00
Tasks: Data Augmentation

Data augmentation is used in machine learning to make the classifier invariant to label-preserving transformations. Usually this invariance is only encouraged implicitly by including a single augmented input during training. However, several works have recently shown that using multiple augmentations per input can improve generalisation or can be used to incorporate invariances more explicitly. In this work, we first empirically compare these recently proposed objectives that differ in whether they rely on explicit or implicit regularisation and at what level of the predictor they encode the invariances. We show that the predictions of the best performing method are also the most similar when compared on different augmentations of the same input. Inspired by this observation, we propose an explicit regulariser that encourages this invariance on the level of individual model predictions. Through extensive experiments on CIFAR-100 and ImageNet we show that this explicit regulariser (i) improves generalisation and (ii) equalises performance differences between all considered objectives. Our results suggest that objectives that encourage invariance on the level of the neural network itself generalise better than those that achieve invariance by averaging predictions of non-invariant models.
PAC-Bayesian Lifelong Learning For Multi-Armed Bandits
Authors: Hamish Flynn, David Reeb, Melih Kandemir, Jan Peters
Date: 2022-03-07 00:00:00
Tasks: Lifelong learning, Multi-Armed Bandits

We present a PAC-Bayesian analysis of lifelong learning. In the lifelong learning problem, a sequence of learning tasks is observed one-at-a-time, and the goal is to transfer information acquired from previous tasks to new learning tasks. We consider the case when each learning task is a multi-armed bandit problem. We derive lower bounds on the expected average reward that would be obtained if a given multi-armed bandit algorithm was run in a new task with a particular prior and for a set number of steps. We propose lifelong learning algorithms that use our new bounds as learning objectives. Our proposed algorithms are evaluated in several lifelong multi-armed bandit problems and are found to perform better than a baseline method that does not use generalisation bounds.
Machine Learning based Anomaly Detection for Smart Shirt: A Systematic Review
Authors: E. C. Nunes
Date: 2022-03-07 00:00:00
Tasks: Anomaly Detection, BIG-bench Machine Learning

In recent years, the popularity and use of Artificial Intelligence (AI) and large investments on theInternet of Medical Things (IoMT) will be common to use products such as smart socks, smartpants, and smart shirts. These products are known as Smart Textile or E-textile, which has theability to monitor and collect signals that our body emits. These signals make it possible to extractanomalous components using Machine Learning (ML) techniques that play an essential role in thisarea. This study presents a Systematic Review of the Literature (SLR) on Anomaly Detection usingML techniques in Smart Shirt. The objectives of the SLR are: (i) to identify what type of anomalythe smart shirt; (ii) what ML techniques are being used; (iii) which datasets are being used; (iv)identify smart shirt or signal acquisition devices; (v) list the performance metrics used to evaluatethe ML model; (vi) the results of the techniques in general; (vii) types of ML algorithms are beingapplied.The SLR selected 11 primary studies published between 2017-2021. The results showed that6 types of anomalies were identified, with the Fall anomaly being the most cited. The Support VectorMachines (SVM) algorithm is most used. Most of the primary studies used public or private datasets.The Hexoskin smart shirt was most cited. The most used metric performance was Accuracy. Onaverage, almost all primary studies presented a result above 90%, and all primary studies used theSupervisioned type of ML.
Neural network approach to reconstructing spectral functions and complex poles of confined particles
Authors: Thibault Lechien, David Dudal
Date: 2022-03-07 00:00:00

Reconstructing spectral functions from propagator data is difficult as solving the analytic continuation problem or applying an inverse integral transformation are ill-conditioned problems. Recent work has proposed using neural networks to solve this problem and has shown promising results, either matching or improving upon the performance of other methods. We generalize this approach by not only reconstructing spectral functions, but also (possible) pairs of complex poles or an infrared (IR) cutoff. We train our network on physically motivated toy functions, examine the reconstruction accuracy and check its robustness to noise. Encouraging results are found on both toy functions and genuine lattice QCD data for the gluon propagator, suggesting that this approach may lead to significant improvements over current state-of-the-art methods.
A deep branching solver for fully nonlinear partial differential equations
Authors: Jiang Yu Nguwi, Guillaume Penent, Nicolas Privault
Date: 2022-03-07 00:00:00
Tasks: Deep Learning

We present a multidimensional deep learning implementation of a stochastic branching algorithm for the numerical solution of fully nonlinear PDEs. This approach is designed to tackle functional nonlinearities involving gradient terms of any orders, by combining the use of neural networks with a Monte Carlo branching algorithm. In comparison with other deep learning PDE solvers, it also allows us to check the consistency of the learned neural network function. Numerical experiments presented show that this algorithm can outperform deep learning approaches based on backward stochastic differential equations or the Galerkin method, and provide solution estimates that are not obtained by those methods in fully nonlinear examples.
Generalized Spectral Clustering for Directed and Undirected Graphs
Authors: Harry Sevi, Matthieu Jonckheere, Argyris Kalogeratos
Date: 2022-03-07 00:00:00
Tasks: Clustering, graph partitioning

Spectral clustering is a popular approach for clustering undirected graphs, but its extension to directed graphs (digraphs) is much more challenging. A typical workaround is to naively symmetrize the adjacency matrix of the directed graph, which can however lead to discarding valuable information carried by edge directionality. In this paper, we present a generalized spectral clustering framework that can address both directed and undirected graphs. Our approach is based on the spectral relaxation of a new functional that we introduce as the generalized Dirichlet energy of a graph function, with respect to an arbitrary positive regularizing measure on the graph edges. We also propose a practical parametrization of the regularizing measure constructed from the iterated powers of the natural random walk on the graph. We present theoretical arguments to explain the efficiency of our framework in the challenging setting of unbalanced classes. Experiments using directed K-NN graphs constructed from real datasets show that our graph partitioning method performs consistently well in all cases, while outperforming existing approaches in most of them.
Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features
Authors: Florian Lux, Ngoc Thang Vu
Date: 2022-03-07 00:00:00
Proceeding: ACL 2022 5
Tasks: Meta-Learning, text-to-speech, Text to Speech

While neural text-to-speech systems perform remarkably well in high-resource scenarios, they cannot be applied to the majority of the over 6,000 spoken languages in the world due to a lack of appropriate training data. In this work, we use embeddings derived from articulatory vectors rather than embeddings derived from phoneme identities to learn phoneme representations that hold across languages. In conjunction with language agnostic meta learning, this enables us to fine-tune a high-quality text-to-speech model on just 30 minutes of data in a previously unseen language spoken by a previously unseen speaker.
Speaker recognition by means of a combination of linear and nonlinear predictive models
Authors: Marcos Faundez-Zanuy
Date: 2022-03-07 00:00:00
Tasks: Speaker Recognition

This paper deals the combination of nonlinear predictive models with classical LPCC parameterization for speaker recognition. It is shown that the combination of both a measure defined over LPCC coefficients and a measure defined over predictive analysis residual signal gives rise to an improvement over the classical method that considers only the LPCC coefficients. If the residual signal is obtained from a linear prediction analysis, the improvement is 2.63% (error rate drops from 6.31% to 3.68%) and if it is computed through a nonlinear predictive neural nets based model, the improvement is 3.68%. An efficient algorithm for reducing the computational burden is also proposed.
A comparative study of several ADPCM schemes with linear and nonlinear prediction
Authors: Oscar Oliva, Marcos Faundez-Zanuy
Date: 2022-03-07 00:00:00
Tasks: Prediction, Quantization

In this paper we compare several ADPCM schemes with nonlinear prediction based on neural nets with the classical ADPCM schemes based on several linear prediction schemes. Main studied variations of the ADPCM scheme with adaptive quantization (2 to 5 bits) are: -forward vs backward -sample adaptive vs block adaptive
Non-linear predictive vector quantization of speech
Authors: Marcos Faundez-Zanuy
Date: 2022-03-07 00:00:00
Tasks: Quantization

In this paper we propose a Non-Linear Predictive Vector quantizer (PVQ) for speech coding, based on Multi-Layer Perceptrons. We also propose a method to evaluate if a quantizer is well designed, and if it exploits the correlation between consecutive outputs. Although the results of the Non-linear PVQ do not improve the results of the non-linear scalar predictor, we check that there is some room for the PVQ improvement.
Bandits Corrupted by Nature: Lower Bounds on Regret and Robust Optimistic Algorithm
Authors: Debabrota Basu, Odalric-Ambrym Maillard, Timothée Mathieu
Date: 2022-03-07 00:00:00

We study the corrupted bandit problem, i.e. a stochastic multi-armed bandit problem with $k$ unknown reward distributions, which are heavy-tailed and corrupted by a history-independent adversary or Nature. To be specific, the reward obtained by playing an arm comes from corresponding heavy-tailed reward distribution with probability $1-\varepsilon \in (0.5,1]$ and an arbitrary corruption distribution of unbounded support with probability $\varepsilon \in [0,0.5)$. First, we provide $\textit{a problem-dependent lower bound on the regret}$ of any corrupted bandit algorithm. The lower bounds indicate that the corrupted bandit problem is harder than the classical stochastic bandit problem with sub-Gaussian or heavy-tail rewards. Following that, we propose a novel UCB-type algorithm for corrupted bandits, namely HubUCB, that builds on Huber's estimator for robust mean estimation. Leveraging a novel concentration inequality of Huber's estimator, we prove that HubUCB achieves a near-optimal regret upper bound. Since computing Huber's estimator has quadratic complexity, we further introduce a sequential version of Huber's estimator that exhibits linear complexity. We leverage this sequential estimator to design SeqHubUCB that enjoys similar regret guarantees while reducing the computational burden. Finally, we experimentally illustrate the efficiency of HubUCB and SeqHubUCB in solving corrupted bandits for different reward distributions and different levels of corruptions.