pwc-archive/papers-with-abstracts huggingface dataset:
Multilevel Stein variational gradient descent with applications to Bayesian inverse problems
Authors: Terrence Alsup, Luca Venturi, Benjamin Peherstorfer
Date: 2021-04-05 00:00:00

This work presents a multilevel variant of Stein variational gradient descent to more efficiently sample from target distributions. The key ingredient is a sequence of distributions with growing fidelity and costs that converges to the target distribution of interest. For example, such a sequence of distributions is given by a hierarchy of ever finer discretization levels of the forward model in Bayesian inverse problems. The proposed multilevel Stein variational gradient descent moves most of the iterations to lower, cheaper levels with the aim of requiring only a few iterations on the higher, more expensive levels when compared to the traditional, single-level Stein variational gradient descent variant that uses the highest-level distribution only. Under certain assumptions, in the mean-field limit, the error of the proposed multilevel Stein method decays by a log factor faster than the error of the single-level counterpart with respect to computational costs. Numerical experiments with Bayesian inverse problems show speedups of more than one order of magnitude of the proposed multilevel Stein method compared to the single-level variant that uses the highest level only.
Optimal Query Complexity of Secure Stochastic Convex Optimization
Authors: Wei Tang, Chien-Ju Ho, Yang Liu
Date: 2021-04-05 00:00:00
Proceeding: NeurIPS 2020 12

We study the secure stochastic convex optimization problem. A learner aims to learn the optimal point of a convex function through sequentially querying a (stochastic) gradient oracle. In the meantime, there exists an adversary who aims to free-ride and infer the learning outcome of the learner from observing the learner's queries. The adversary observes only the points of the queries but not the feedback from the oracle. The goal of the learner is to optimize the accuracy, i.e., obtaining an accurate estimate of the optimal point, while securing her privacy, i.e., making it difficult for the adversary to infer the optimal point. We formally quantify this tradeoff between learner's accuracy and privacy and characterize the lower and upper bounds on the learner's query complexity as a function of desired levels of accuracy and privacy. For the analysis of lower bounds, we provide a general template based on information theoretical analysis and then tailor the template to several families of problems, including stochastic convex optimization and (noisy) binary search. We also present a generic secure learning protocol that achieves the matching upper bound up to logarithmic factors.
DexDeepFM: Ensemble Diversity Enhanced Extreme Deep Factorization Machine Model
Authors: Ling Chen, Hongyu Shi
Date: 2021-04-05 00:00:00
Tasks: Diversity

Predicting user positive response (e.g., purchases and clicks) probability is a critical task in Web applications. To identify predictive features from raw data, the state-of-the-art extreme deep factorization machine model (xDeepFM) introduces a new interaction network to leverage feature interactions at the vector-wise level explicitly. However, since each hidden layer in the interaction network is a collection of feature maps, it can be viewed essentially as an ensemble of different feature maps. In this case, only using a single objective to minimize the prediction loss may lead to overfitting and generate correlated errors. In this paper, an ensemble diversity enhanced extreme deep factorization machine model (DexDeepFM) is proposed, which designs the ensemble diversity measure in each hidden layer and considers both ensemble diversity and prediction accuracy in the objective function. In addition, the attention mechanism is introduced to discriminate the importance of ensemble diversity measures with different feature interaction orders. Extensive experiments on three public real-world datasets are conducted to show the effectiveness of the proposed model.
Conformal testing in a binary model situation
Authors: Vladimir Vovk
Date: 2021-04-05 00:00:00
Tasks: Conformal Prediction, model

Conformal testing is a way of testing the IID assumption based on conformal prediction. The topic of this note is computational evaluation of the performance of conformal testing in a model situation in which IID binary observations generated from a Bernoulli distribution are followed by IID binary observations generated from another Bernoulli distribution, with the parameters of the distributions and changepoint unknown. Existing conformal test martingales can be used for this task and work well in simple cases, but their efficiency can be improved greatly.
FedPandemic: A Cross-Device Federated Learning Approach Towards Elementary Prognosis of Diseases During a Pandemic
Authors: Aman Priyanshu, Rakshit Naidu
Date: 2021-04-05 00:00:00
Tasks: Federated Learning, Prognosis, Retrieval

The amount of data, manpower and capital required to understand, evaluate and agree on a group of symptoms for the elementary prognosis of pandemic diseases is enormous. In this paper, we present FedPandemic, a novel noise implementation algorithm integrated with cross-device Federated learning for Elementary symptom prognosis during a pandemic, taking COVID-19 as a case study. Our results display consistency and enhance robustness in recovering the common symptoms displayed by the disease, paving a faster and cheaper path towards symptom retrieval while also preserving the privacy of patient's symptoms via Federated learning.
Rethinking Perturbations in Encoder-Decoders for Fast Training
Authors: Sho Takase, Shun Kiyono
Date: 2021-04-05 00:00:00
Proceeding: NAACL 2021 4
Tasks: Machine Translation, Text Summarization

We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbations (Sato et al., 2019) as perturbations but these methods require considerable computational time. Thus, this study addresses the question of whether these approaches are efficient enough for training time. We compare several perturbations in sequence-to-sequence problems with respect to computational time. Experimental results show that the simple techniques such as word dropout (Gal and Ghahramani, 2016) and random replacement of input tokens achieve comparable (or better) scores to the recently proposed perturbations, even though these simple methods are faster. Our code is publicly available at https://github.com/takase/rethink_perturbations.
Improving the Expressive Power of Graph Neural Network with Tinhofer Algorithm
Authors: Alan J. X. Guo, Qing-Hu Hou, Ou wu
Date: 2021-04-05 00:00:00
Tasks: Graph Neural Network

In recent years, Graph Neural Network (GNN) has bloomly progressed for its power in processing graph-based data. Most GNNs follow a message passing scheme, and their expressive power is mathematically limited by the discriminative ability of the Weisfeiler-Lehman (WL) test. Following Tinhofer's research on compact graphs, we propose a variation of the message passing scheme, called the Weisfeiler-Lehman-Tinhofer GNN (WLT-GNN), that theoretically breaks through the limitation of the WL test. In addition, we conduct comparative experiments and ablation studies on several well-known datasets. The results show that the proposed methods have comparable performances and better expressive power on these datasets.
Stopping Criterion for Active Learning Based on Error Stability
Authors: Hideaki Ishibashi, Hideitsu Hino
Date: 2021-04-05 00:00:00
Tasks: Active Learning

Active learning is a framework for supervised learning to improve the predictive performance by adaptively annotating a small number of samples. To realize efficient active learning, both an acquisition function that determines the next datum and a stopping criterion that determines when to stop learning should be considered. In this study, we propose a stopping criterion based on error stability, which guarantees that the change in generalization error upon adding a new sample is bounded by the annotation cost and can be applied to any Bayesian active learning. We demonstrate that the proposed criterion stops active learning at the appropriate timing for various learning models and real datasets.
Model Compression for Dynamic Forecast Combination
Authors: Vitor Cerqueira, Luis Torgo, Carlos Soares, Albert Bifet
Date: 2021-04-05 00:00:00
Tasks: model, Model Compression, Time Series, Time Series Analysis, Time Series Forecasting

The predictive advantage of combining several different predictive models is widely accepted. Particularly in time series forecasting problems, this combination is often dynamic to cope with potential non-stationary sources of variation present in the data. Despite their superior predictive performance, ensemble methods entail two main limitations: high computational costs and lack of transparency. These issues often preclude the deployment of such approaches, in favour of simpler yet more efficient and reliable ones. In this paper, we leverage the idea of model compression to address this problem in time series forecasting tasks. Model compression approaches have been mostly unexplored for forecasting. Their application in time series is challenging due to the evolving nature of the data. Further, while the literature focuses on neural networks, we apply model compression to distinct types of methods. In an extensive set of experiments, we show that compressing dynamic forecasting ensembles into an individual model leads to a comparable predictive performance and a drastic reduction in computational costs. Further, the compressed individual model with best average rank is a rule-based regression model. Thus, model compression also leads to benefits in terms of model interpretability. The experiments carried in this paper are fully reproducible.
Modeling Gate-Level Abstraction Hierarchy Using Graph Convolutional Neural Networks to Predict Functional De-Rating Factors
Authors: Aneesh Balakrishnan, Thomas Lange, Maximilien Glorieux, Dan Alexandrescu, Maksim Jenihhin
Date: 2021-04-05 00:00:00

The paper is proposing a methodology for modeling a gate-level netlist using a Graph Convolutional Network (GCN). The model predicts the overall functional de-rating factors of sequential elements of a given circuit. In the preliminary phase of the work, the important goal is making a GCN which able to take a gate-level netlist as input information after transforming it into the Probabilistic Bayesian Graph in the form of Graph Modeling Language (GML). This part enables the GCN to learn the structural information of netlist in graph domains. In the second phase of the work, the modeled GCN trained with the a functional de-rating factor of a very low number of individual sequential elements (flip-flops). The third phase includes understanding of GCN models accuracy to model an arbitrary circuit netlist. The designed model was validated for two circuits. One is the IEEE 754 standard double precision floating point adder and the second one is the 10-Gigabit Ethernet MAC IEEE802.3 standard. The predicted results compared to the standard fault injection campaign results of the error called Single EventUpset (SEU). The validated results are graphically pictured in the form of the histogram and sorted probabilities and evaluated with the Confidence Interval (CI) metric between the predicted and simulated fault injection results.
Frequency Estimation Under Multiparty Differential Privacy: One-shot and Streaming
Authors: Ziyue Huang, Yuan Qiu, Ke Yi, Graham Cormode
Date: 2021-04-05 00:00:00

We study the fundamental problem of frequency estimation under both privacy and communication constraints, where the data is distributed among $k$ parties. We consider two application scenarios: (1) one-shot, where the data is static and the aggregator conducts a one-time computation; and (2) streaming, where each party receives a stream of items over time and the aggregator continuously monitors the frequencies. We adopt the model of multiparty differential privacy (MDP), which is more general than local differential privacy (LDP) and (centralized) differential privacy. Our protocols achieve optimality (up to logarithmic factors) permissible by the more stringent of the two constraints. In particular, when specialized to the $\varepsilon$-LDP model, our protocol achieves an error of $\sqrt{k}/(e^{\Theta(\varepsilon)}-1)$ using $O(k\max\{ \varepsilon, \frac{1}{\varepsilon} \})$ bits of communication and $O(k \log u)$ bits of public randomness, where $u$ is the size of the domain.
CCSNet: a deep learning modeling suite for CO$_2$ storage
Authors: Gege Wen, Catherine Hay, Sally M. Benson
Date: 2021-04-05 00:00:00
Tasks: Computational Efficiency, Deep Learning

Numerical simulation is an essential tool for many applications involving subsurface flow and transport, yet often suffers from computational challenges due to the multi-physics nature, highly non-linear governing equations, inherent parameter uncertainties, and the need for high spatial resolutions to capture multi-scale heterogeneity. We developed CCSNet, a general-purpose deep-learning modeling suite that can act as an alternative to conventional numerical simulators for carbon capture and storage (CCS) problems where CO$_2$ is injected into saline aquifers in 2d-radial systems. CCSNet consists of a sequence of deep learning models producing all the outputs that a numerical simulator typically provides, including saturation distributions, pressure buildup, dry-out, fluid densities, mass balance, solubility trapping, and sweep efficiency. The results are 10$^3$ to 10$^4$ times faster than conventional numerical simulators. As an application of CCSNet illustrating the value of its high computational efficiency, we developed rigorous estimation techniques for the sweep efficiency and solubility trapping.
Application of Neural Network Algorithm in Propylene Distillation
Authors: Jinwei Lu, Ningrui Zhao
Date: 2021-04-05 00:00:00
Tasks: Self-Learning

Artificial neural network modeling does not need to consider the mechanism. It can map the implicit relationship between input and output and predict the performance of the system well. At the same time, it has the advantages of self-learning ability and high fault tolerance. The gas-liquid two phases in the rectification tower conduct interphase heat and mass transfer through countercurrent contact. The functional relationship between the product concentration at the top and bottom of the tower and the process parameters is extremely complex. The functional relationship can be accurately controlled by artificial neural network algorithms. The key components of the propylene distillation tower are the propane concentration at the top of the tower and the propylene concentration at the bottom of the tower. Accurate measurement of them plays a key role in increasing propylene yield in ethylene production enterprises. This article mainly introduces the neural network model and its application in the propylene distillation tower.
Optimal Sampling Gaps for Adaptive Submodular Maximization
Authors: Shaojie Tang, Jing Yuan
Date: 2021-04-05 00:00:00
Tasks: Active Learning, BIG-bench Machine Learning, Marketing

Running machine learning algorithms on large and rapidly growing volumes of data is often computationally expensive, one common trick to reduce the size of a data set, and thus reduce the computational cost of machine learning algorithms, is \emph{probability sampling}. It creates a sampled data set by including each data point from the original data set with a known probability. Although the benefit of running machine learning algorithms on the reduced data set is obvious, one major concern is that the performance of the solution obtained from samples might be much worse than that of the optimal solution when using the full data set. In this paper, we examine the performance loss caused by probability sampling in the context of adaptive submodular maximization. We consider a simple probability sampling method which selects each data point with probability at least $r\in[0,1]$. If we set $r=1$, our problem reduces to finding a solution based on the original full data set. We define sampling gap as the largest ratio between the optimal solution obtained from the full data set and the optimal solution obtained from the samples, over independence systems. Our main contribution is to show that if the sampling probability of each data point is at least $r$ and the utility function is policywise submodular, then the sampling gap is both upper bounded and lower bounded by $1/r$. We show that the property of policywise submodular can be found in a wide range of real-world applications, including pool-based active learning and adaptive viral marketing.
Graph Generative Models for Fast Detector Simulations in High Energy Physics
Authors: Ali Hariri, Darya Dyachkova, Sergei Gleyzer
Date: 2021-04-05 00:00:00
Tasks: Vocal Bursts Intensity Prediction

Accurate and fast simulation of particle physics processes is crucial for the high-energy physics community. Simulating particle interactions with detectors is both time consuming and computationally expensive. With the proton-proton collision energy of 13 TeV, the Large Hadron Collider is uniquely positioned to detect and measure the rare phenomena that can shape our knowledge of new interactions. The High-Luminosity Large Hadron Collider (HL-LHC) upgrade will put a significant strain on the computing infrastructure due to increased event rate and levels of pile-up. Simulation of high-energy physics collisions needs to be significantly faster without sacrificing the physics accuracy. Machine learning approaches can offer faster solutions, while maintaining a high level of fidelity. We discuss a graph generative model that provides effective reconstruction of LHC events, paving the way for full detector level fast simulation for HL-LHC.
Analyzing Flight Delay Prediction Under Concept Drift
Authors: Lucas Giusti, Leonardo Carvalho, Antonio Tadeu Gomes, Rafaelli Coutinho, Jorge Soares, Eduardo Ogasawara
Date: 2021-04-05 00:00:00
Tasks: Prediction

Flight delays impose challenges that impact any flight transportation system. Predicting when they are going to occur is an important way to mitigate this issue. However, the behavior of the flight delay system varies through time. This phenomenon is known in predictive analytics as concept drift. This paper investigates the prediction performance of different drift handling strategies in aviation under different scales (models trained from flights related to a single airport or the entire flight system). Specifically, two research questions were proposed and answered: (i) How do drift handling strategies influence the prediction performance of delays? (ii) Do different scales change the results of drift handling strategies? In our analysis, drift handling strategies are relevant, and their impacts vary according to scale and machine learning models used.
Quaternion Factorization Machines: A Lightweight Solution to Intricate Feature Interaction Modelling
Authors: Tong Chen, Hongzhi Yin, Xiangliang Zhang, Zi Huang, Yang Wang, Meng Wang
Date: 2021-04-05 00:00:00
Tasks: Feature Engineering

As a well-established approach, factorization machine (FM) is capable of automatically learning high-order interactions among features to make predictions without the need for manual feature engineering. With the prominent development of deep neural networks (DNNs), there is a recent and ongoing trend of enhancing the expressiveness of FM-based models with DNNs. However, though better results are obtained with DNN-based FM variants, such performance gain is paid off by an enormous amount (usually millions) of excessive model parameters on top of the plain FM. Consequently, the heavy parameterization impedes the real-life practicality of those deep models, especially efficient deployment on resource-constrained IoT and edge devices. In this paper, we move beyond the traditional real space where most deep FM-based models are defined, and seek solutions from quaternion representations within the hypercomplex space. Specifically, we propose the quaternion factorization machine (QFM) and quaternion neural factorization machine (QNFM), which are two novel lightweight and memory-efficient quaternion-valued models for sparse predictive analytics. By introducing a brand new take on FM-based models with the notion of quaternion algebra, our models not only enable expressive inter-component feature interactions, but also significantly reduce the parameter size due to lower degrees of freedom in the hypercomplex Hamilton product compared with real-valued matrix multiplication. Extensive experimental results on three large-scale datasets demonstrate that QFM achieves 4.36% performance improvement over the plain FM without introducing any extra parameters, while QNFM outperforms all baselines with up to two magnitudes' parameter size reduction in comparison to state-of-the-art peer methods.
Detecting of multi-modality in probabilistic regression models
Authors: Andrew Polar, Michael Poluektov
Date: 2021-04-04 00:00:00
Tasks: Additive models, Descriptive, Ensemble Learning, regression

This paper focuses on building of models of stochastic systems with aleatoric uncertainty. The nature of the considered systems is such that the identical inputs can result in different outputs, i.e. the output is a random variable. The suggested in this paper algorithm targets an identification of multi-modal properties of the output distributions, even when they depend on the inputs and vary significantly throughout the dataset. This ability of the suggested method to recognise complex and not only bell-shaped distributions follows from its construction and is backed up by provided experimental results. In general, the suggested method belongs to the category of boosted ensemble learning techniques, where the single deterministic component can be an arbitrarily-chosen regression model. The algorithm does not require any special properties of the chosen regression model, other than having descriptive capabilities with some expected accuracy for the training data type.
Identification of Nonlinear Dynamic Systems Using Type-2 Fuzzy Neural Networks -- A Novel Learning Algorithm and a Comparative Study
Authors: Erkan Kayacan, Erdal Kayacan, Mojtaba Ahmadieh Khanesar
Date: 2021-04-04 00:00:00

In order to achieve faster and more robust convergence (especially under noisy working environments), a sliding mode theory-based learning algorithm has been proposed to tune both the premise and consequent parts of type-2 fuzzy neural networks in this paper. Differently from recent studies, where sliding mode control theory-based rules are proposed for only the consequent part of the network, the developed algorithm applies fully sliding mode parameter update rules for both the premise and consequent parts of the type-2 fuzzy neural networks. In addition, the responsible parameter for sharing the contributions of the lower and upper parts of the type-2 fuzzy membership functions is also tuned. Moreover, the learning rate of the network is updated during the online training. The stability of the proposed learning algorithm has been proved by using an appropriate Lyapunov function. Several comparisons have been realized and shown that the proposed algorithm has faster convergence speed than the existing methods such as gradient-based and swarm intelligence-based methods. Moreover, the proposed learning algorithm has a closed form, and it is easier to implement than the other existing methods.