pwc-archive/papers-with-abstracts huggingface dataset:
EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models
Authors: Jiangsu Du, Ziming Liu, Jiarui Fang, Shenggui Li, Yongbin Li, Yutong Lu, Yang You
Date: 2022-09-06 00:00:00
Tasks: Blocking, GPU

Large transformer models display promising performance on a wide range of natural language processing (NLP) tasks. Although the AI community has expanded the model scale to the trillion parameter level, the practical deployment of 10-100 billion parameter models is still uncertain due to the latency, throughput, and memory constraints. In this paper, we proposed EnergonAI to solve the challenges of the efficient deployment of 10-100 billion parameter transformer models on single- or multi-GPU systems. EnergonAI adopts a hierarchy-controller system architecture to coordinate multiple devices and efficiently support different parallel patterns. It delegates the execution of sub-models to multiple workers in the single-controller style and applies tensor parallelism and pipeline parallelism among the workers in a multi-controller style. Upon the novel architecture, we propose three techniques, i.e. non-blocking pipeline parallelism, distributed redundant computation elimination, and peer memory pooling. EnergonAI enables the users to program complex parallel code the same as a serial one. Compared with the FasterTransformer, we have proven that EnergonAI has superior performance on latency and throughput. In our experiments, EnergonAI can achieve 37% latency reduction in tensor parallelism, 10% scalability improvement in pipeline parallelism, and it improves the model scale inferred on a single GPU by using a larger heterogeneous memory space at cost of limited performance reduction.
Wavelet-based Loss for High-frequency Interface Dynamics
Authors: Lukas Prantl, Jan Bender, Tassilo Kugelstadt, Nils Thuerey
Date: 2022-09-06 00:00:00
Tasks: Physical Simulations, Vocal Bursts Intensity Prediction

Generating highly detailed, complex data is a long-standing and frequently considered problem in the machine learning field. However, developing detail-aware generators remains an challenging and open problem. Generative adversarial networks are the basis of many state-of-the-art methods. However, they introduce a second network to be trained as a loss function, making the interpretation of the learned functions much more difficult. As an alternative, we present a new method based on a wavelet loss formulation, which remains transparent in terms of what is optimized. The wavelet-based loss function is used to overcome the limitations of conventional distance metrics, such as L1 or L2 distances, when it comes to generate data with high-frequency details. We show that our method can successfully reconstruct high-frequency details in an illustrative synthetic test case. Additionally, we evaluate the performance when applied to more complex surfaces based on physical simulations. Taking a roughly approximated simulation as input, our method infers corresponding spatial details while taking into account how they evolve. We consider this problem in terms of spatial and temporal frequencies, and leverage generative networks trained with our wavelet loss to learn the desired spatio-temporal signal for the surface dynamics. We test the capabilities of our method with a set of synthetic wave function tests and complex 2D and 3D dynamics of elasto-plastic materials.
Rates of Convergence for Regression with the Graph Poly-Laplacian
Authors: Nicolás García Trillos, Ryan Murray, Matthew Thorpe
Date: 2022-09-06 00:00:00
Tasks: regression

In the (special) smoothing spline problem one considers a variational problem with a quadratic data fidelity penalty and Laplacian regularisation. Higher order regularity can be obtained via replacing the Laplacian regulariser with a poly-Laplacian regulariser. The methodology is readily adapted to graphs and here we consider graph poly-Laplacian regularisation in a fully supervised, non-parametric, noise corrupted, regression problem. In particular, given a dataset $\{x_i\}_{i=1}^n$ and a set of noisy labels $\{y_i\}_{i=1}^n\subset\mathbb{R}$ we let $u_n:\{x_i\}_{i=1}^n\to\mathbb{R}$ be the minimiser of an energy which consists of a data fidelity term and an appropriately scaled graph poly-Laplacian term. When $y_i = g(x_i)+\xi_i$, for iid noise $\xi_i$, and using the geometric random graph, we identify (with high probability) the rate of convergence of $u_n$ to $g$ in the large data limit $n\to\infty$. Furthermore, our rate, up to logarithms, coincides with the known rate of convergence in the usual smoothing spline model.
Faster federated optimization under second-order similarity
Authors: Ahmed Khaled, Chi Jin
Date: 2022-09-06 00:00:00
Tasks: Federated Learning

Federated learning (FL) is a subfield of machine learning where multiple clients try to collaboratively learn a model over a network under communication constraints. We consider finite-sum federated optimization under a second-order function similarity condition and strong convexity, and propose two new algorithms: SVRP and Catalyzed SVRP. This second-order similarity condition has grown popular recently, and is satisfied in many applications including distributed statistical learning and differentially private empirical risk minimization. The first algorithm, SVRP, combines approximate stochastic proximal point evaluations, client sampling, and variance reduction. We show that SVRP is communication efficient and achieves superior performance to many existing algorithms when function similarity is high enough. Our second algorithm, Catalyzed SVRP, is a Catalyst-accelerated variant of SVRP that achieves even better performance and uniformly improves upon existing algorithms for federated optimization under second-order similarity and strong convexity. In the course of analyzing these algorithms, we provide a new analysis of the Stochastic Proximal Point Method (SPPM) that might be of independent interest. Our analysis of SPPM is simple, allows for approximate proximal point evaluations, does not require any smoothness assumptions, and shows a clear benefit in communication complexity over ordinary distributed stochastic gradient descent.
Making the black-box brighter: interpreting machine learning algorithm for forecasting drilling accidents
Authors: Ekaterina Gurina, Nikita Klyuchnikov, Ksenia Antipova, Dmitry Koroteev
Date: 2022-09-06 00:00:00

We present an approach for interpreting a black-box alarming system for forecasting accidents and anomalies during the drilling of oil and gas wells. The interpretation methodology aims to explain the local behavior of the accident predictive model to drilling engineers. The explanatory model uses Shapley additive explanations analysis of features, obtained through Bag-of-features representation of telemetry logs used during the drilling accident forecasting phase. Validation shows that the explanatory model has 15% precision at 70% recall, and overcomes the metric values of a random baseline and multi-head attention neural network. These results justify that the developed explanatory model is better aligned with explanations of drilling engineers, than the state-of-the-art method. The joint performance of explanatory and Bag-of-features models allows drilling engineers to understand the logic behind the system decisions at the particular moment, pay attention to highlighted telemetry regions, and correspondingly, increase the trust level in the accident forecasting alarms.
Machine Learning For Classification Of Antithetical Emotional States
Authors: Jeevanshi Sharma, Rajat Maheshwari, Yusuf Uzzaman Khan
Date: 2022-09-06 00:00:00
Tasks: Classification, EEG, Electroencephalogram (EEG), Emotion Classification

Emotion Classification through EEG signals has achieved many advancements. However, the problems like lack of data and learning the important features and patterns have always been areas with scope for improvement both computationally and in prediction accuracy. This works analyses the baseline machine learning classifiers' performance on DEAP Dataset along with a tabular learning approach that provided state-of-the-art comparable results leveraging the performance boost due to its deep learning architecture without deploying heavy neural networks.
The mpEDMD Algorithm for Data-Driven Computations of Measure-Preserving Dynamical Systems
Authors: Matthew J. Colbrook
Date: 2022-09-06 00:00:00

Koopman operators globally linearize nonlinear dynamical systems and their spectral information is a powerful tool for the analysis and decomposition of nonlinear dynamical systems. However, Koopman operators are infinite-dimensional, and computing their spectral information is a considerable challenge. We introduce measure-preserving extended dynamic mode decomposition ($\texttt{mpEDMD}$), the first truncation method whose eigendecomposition converges to the spectral quantities of Koopman operators for general measure-preserving dynamical systems. $\texttt{mpEDMD}$ is a data-driven algorithm based on an orthogonal Procrustes problem that enforces measure-preserving truncations of Koopman operators using a general dictionary of observables. It is flexible and easy to use with any pre-existing DMD-type method, and with different types of data. We prove convergence of $\texttt{mpEDMD}$ for projection-valued and scalar-valued spectral measures, spectra, and Koopman mode decompositions. For the case of delay embedding (Krylov subspaces), our results include the first convergence rates of the approximation of spectral measures as the size of the dictionary increases. We demonstrate $\texttt{mpEDMD}$ on a range of challenging examples, its increased robustness to noise compared with other DMD-type methods, and its ability to capture the energy conservation and cascade of experimental measurements of a turbulent boundary layer flow with Reynolds number $> 6\times 10^4$ and state-space dimension $>10^5$.
Multi-Armed Bandits with Self-Information Rewards
Authors: Nir Weinberger, Michal Yemini
Date: 2022-09-06 00:00:00
Tasks: Multi-Armed Bandits

This paper introduces the informational multi-armed bandit (IMAB) model in which at each round, a player chooses an arm, observes a symbol, and receives an unobserved reward in the form of the symbol's self-information. Thus, the expected reward of an arm is the Shannon entropy of the probability mass function of the source that generates its symbols. The player aims to maximize the expected total reward associated with the entropy values of the arms played. Under the assumption that the alphabet size is known, two UCB-based algorithms are proposed for the IMAB model which consider the biases of the plug-in entropy estimator. The first algorithm optimistically corrects the bias term in the entropy estimation. The second algorithm relies on data-dependent confidence intervals that adapt to sources with small entropy values. Performance guarantees are provided by upper bounding the expected regret of each of the algorithms. Furthermore, in the Bernoulli case, the asymptotic behavior of these algorithms is compared to the Lai-Robbins lower bound for the pseudo regret. Additionally, under the assumption that the \textit{exact} alphabet size is unknown, and instead the player only knows a loose upper bound on it, a UCB-based algorithm is proposed, in which the player aims to reduce the regret caused by the unknown alphabet size in a finite time regime. Numerical results illustrating the expected regret of the algorithms presented in the paper are provided.
What to Prune and What Not to Prune at Initialization
Authors: Maham Haroon
Date: 2022-09-06 00:00:00
Tasks: Evolutionary Algorithms

Post-training dropout based approaches achieve high sparsity and are well established means of deciphering problems relating to computational cost and overfitting in Neural Network architectures. Contrastingly, pruning at initialization is still far behind. Initialization pruning is more efficacious when it comes to scaling computation cost of the network. Furthermore, it handles overfitting just as well as post training dropout. In approbation of the above reasons, the paper presents two approaches to prune at initialization. The goal is to achieve higher sparsity while preserving performance. 1) K-starts, begins with k random p-sparse matrices at initialization. In the first couple of epochs the network then determines the "fittest" of these p-sparse matrices in an attempt to find the "lottery ticket" p-sparse network. The approach is adopted from how evolutionary algorithms find the best individual. Depending on the Neural Network architecture, fitness criteria can be based on magnitude of network weights, magnitude of gradient accumulation over an epoch or a combination of both. 2) Dissipating gradients approach, aims at eliminating weights that remain within a fraction of their initial value during the first couple of epochs. Removing weights in this manner despite their magnitude best preserves performance of the network. Contrarily, the approach also takes the most epochs to achieve higher sparsity. 3) Combination of dissipating gradients and kstarts outperforms either methods and random dropout consistently. The benefits of using the provided pertaining approaches are: 1) They do not require specific knowledge of the classification task, fixing of dropout threshold or regularization parameters 2) Retraining of the model is neither necessary nor affects the performance of the p-sparse network.
Bayesian Neural Network Inference via Implicit Models and the Posterior Predictive Distribution
Authors: Joel Janek Dabrowski, Daniel Edward Pagendam
Date: 2022-09-06 00:00:00
Tasks: Bayesian Inference, Density Ratio Estimation, Uncertainty Quantification, Variational Inference

We propose a novel approach to perform approximate Bayesian inference in complex models such as Bayesian neural networks. The approach is more scalable to large data than Markov Chain Monte Carlo, it embraces more expressive models than Variational Inference, and it does not rely on adversarial training (or density ratio estimation). We adopt the recent approach of constructing two models: (1) a primary model, tasked with performing regression or classification; and (2) a secondary, expressive (e.g. implicit) model that defines an approximate posterior distribution over the parameters of the primary model. However, we optimise the parameters of the posterior model via gradient descent according to a Monte Carlo estimate of the posterior predictive distribution -- which is our only approximation (other than the posterior model). Only a likelihood needs to be specified, which can take various forms such as loss functions and synthetic likelihoods, thus providing a form of a likelihood-free approach. Furthermore, we formulate the approach such that the posterior samples can either be independent of, or conditionally dependent upon the inputs to the primary model. The latter approach is shown to be capable of increasing the apparent complexity of the primary model. We see this being useful in applications such as surrogate and physics-based models. To promote how the Bayesian paradigm offers more than just uncertainty quantification, we demonstrate: uncertainty quantification, multi-modality, as well as an application with a recent deep forecasting neural network architecture.
A variational neural network approach for glacier modelling with nonlinear rheology
Authors: Tiangang Cui, Zhongjian Wang, Zhiwen Zhang
Date: 2022-09-05 00:00:00

In this paper, we propose a mesh-free method to solve full stokes equation which models the glacier movement with nonlinear rheology. Our approach is inspired by the Deep-Ritz method proposed in [12]. We first formulate the solution of non-Newtonian ice flow model into the minimizer of a variational integral with boundary constraints. The solution is then approximated by a deep neural network whose loss function is the variational integral plus soft constraint from the mixed boundary conditions. Instead of introducing mesh grids or basis functions to evaluate the loss function, our method only requires uniform samplers of the domain and boundaries. To address instability in real-world scaling, we re-normalize the input of the network at the first layer and balance the regularizing factors for each individual boundary. Finally, we illustrate the performance of our method by several numerical experiments, including a 2D model with analytical solution, Arolla glacier model with real scaling and a 3D model with periodic boundary conditions. Numerical results show that our proposed method is efficient in solving the non-Newtonian mechanics arising from glacier modeling with nonlinear rheology.
GRASP: A Goodness-of-Fit Test for Classification Learning
Authors: Adel Javanmard, Mohammad Mehrabi
Date: 2022-09-05 00:00:00
Tasks: Classification

Performance of classifiers is often measured in terms of average accuracy on test data. Despite being a standard measure, average accuracy fails in characterizing the fit of the model to the underlying conditional law of labels given the features vector ($Y|X$), e.g. due to model misspecification, over fitting, and high-dimensionality. In this paper, we consider the fundamental problem of assessing the goodness-of-fit for a general binary classifier. Our framework does not make any parametric assumption on the conditional law $Y|X$, and treats that as a black box oracle model which can be accessed only through queries. We formulate the goodness-of-fit assessment problem as a tolerance hypothesis testing of the form \[ H_0: \mathbb{E}\Big[D_f\Big({\sf Bern}(\eta(X))\|{\sf Bern}(\hat{\eta}(X))\Big)\Big]\leq \tau\,, \] where $D_f$ represents an $f$-divergence function, and $\eta(x)$, $\hat{\eta}(x)$ respectively denote the true and an estimate likelihood for a feature vector $x$ admitting a positive label. We propose a novel test, called \grasp for testing $H_0$, which works in finite sample settings, no matter the features (distribution-free). We also propose model-X \grasp designed for model-X settings where the joint distribution of the features vector is known. Model-X \grasp uses this distributional information to achieve better power. We evaluate the performance of our tests through extensive numerical experiments.
Applying Machine Learning to Life Insurance: some knowledge sharing to master it
Authors: Antoine Chancel, Laura Bradier, Antoine Ly, Razvan Ionescu, Laurene Martin, Marguerite Sauce
Date: 2022-09-05 00:00:00

Machine Learning permeates many industries, which brings new source of benefits for companies. However within the life insurance industry, Machine Learning is not widely used in practice as over the past years statistical models have shown their efficiency for risk assessment. Thus insurers may face difficulties to assess the value of the artificial intelligence. Focusing on the modification of the life insurance industry over time highlights the stake of using Machine Learning for insurers and benefits that it can bring by unleashing data value. This paper reviews traditional actuarial methodologies for survival modeling and extends them with Machine Learning techniques. It points out differences with regular machine learning models and emphasizes importance of specific implementations to face censored data with machine learning models family. In complement to this article, a Python library has been developed. Different open-source Machine Learning algorithms have been adjusted to adapt the specificities of life insurance data, namely censoring and truncation. Such models can be easily applied from this SCOR library to accurately model life insurance risks.
Full Kullback-Leibler-Divergence Loss for Hyperparameter-free Label Distribution Learning
Authors: Maurice Günder, Nico Piatkowski, Christian Bauckhage
Date: 2022-09-05 00:00:00
Tasks: Age Estimation, regression

The concept of Label Distribution Learning (LDL) is a technique to stabilize classification and regression problems with ambiguous and/or imbalanced labels. A prototypical use-case of LDL is human age estimation based on profile images. Regarding this regression problem, a so called Deep Label Distribution Learning (DLDL) method has been developed. The main idea is the joint regression of the label distribution and its expectation value. However, the original DLDL method uses loss components with different mathematical motivation and, thus, different scales, which is why the use of a hyperparameter becomes necessary. In this work, we introduce a loss function for DLDL whose components are completely defined by Kullback-Leibler (KL) divergences and, thus, are directly comparable to each other without the need of additional hyperparameters. It generalizes the concept of DLDL with regard to further use-cases, in particular for multi-dimensional or multi-scale distribution learning tasks.
Advancing Reacting Flow Simulations with Data-Driven Models
Authors: Kamila Zdybał, Giuseppe D'Alessio, Gianmarco Aversano, Mohammad Rafi Malik, Axel Coussement, James C. Sutherland, Alessandro Parente
Date: 2022-09-05 00:00:00

The use of machine learning algorithms to predict behaviors of complex systems is booming. However, the key to an effective use of machine learning tools in multi-physics problems, including combustion, is to couple them to physical and computer models. The performance of these tools is enhanced if all the prior knowledge and the physical constraints are embodied. In other words, the scientific method must be adapted to bring machine learning into the picture, and make the best use of the massive amount of data we have produced, thanks to the advances in numerical computing. The present chapter reviews some of the open opportunities for the application of data-driven reduced-order modeling of combustion systems. Examples of feature extraction in turbulent combustion data, empirical low-dimensional manifold (ELDM) identification, classification, regression, and reduced-order modeling are provided.
Visualization Of Class Activation Maps To Explain AI Classification Of Network Packet Captures
Authors: Igor Cherepanov, Alex Ulmer, Jonathan Geraldi Joewono, Jörn Kohlhammer
Date: 2022-09-05 00:00:00
Tasks: Classification

The classification of internet traffic has become increasingly important due to the rapid growth of today's networks and applications. The number of connections and the addition of new applications in our networks causes a vast amount of log data and complicates the search for common patterns by experts. Finding such patterns among specific classes of applications is necessary to fulfill various requirements in network analytics. Deep learning methods provide both feature extraction and classification from data in a single system. However, these networks are very complex and are used as black-box models, which weakens the experts' trust in the classifications. Moreover, by using them as a black-box, new knowledge cannot be obtained from the model predictions despite their excellent performance. Therefore, the explainability of the classifications is crucial. Besides increasing trust, the explanation can be used for model evaluation gaining new insights from the data and improving the model. In this paper, we present a visual interactive tool that combines the classification of network data with an explanation technique to form an interface between experts, algorithms, and data.
Online Decision Making for Trading Wind Energy
Authors: Miguel Angel Muñoz, Pierre Pinson, Jalal Kazempour
Date: 2022-09-05 00:00:00
Tasks: Decision Making

We propose and develop a new algorithm for trading wind energy in electricity markets, within an online learning and optimization framework. In particular, we combine a component-wise adaptive variant of the gradient descent algorithm with recent advances in the feature-driven newsvendor model. This results in an online offering approach capable of leveraging data-rich environments, while adapting to the nonstationary characteristics of energy generation and electricity markets, also with a minimal computational burden. The performance of our approach is analyzed based on several numerical experiments, showing both better adaptability to nonstationary uncertain parameters and significant economic gains.