pwc-archive/papers-with-abstracts huggingface dataset:
Neural Tangent Kernel of Matrix Product States: Convergence and Applications
Authors: Erdong Guo, David Draper
Date: 2021-11-28 00:00:00
Tasks: regression

In this work, we study the Neural Tangent Kernel (NTK) of Matrix Product States (MPS) and the convergence of its NTK in the infinite bond dimensional limit. We prove that the NTK of MPS asymptotically converges to a constant matrix during the gradient descent (training) process (and also the initialization phase) as the bond dimensions of MPS go to infinity by the observation that the variation of the tensors in MPS asymptotically goes to zero during training in the infinite limit. By showing the positive-definiteness of the NTK of MPS, the convergence of MPS during the training in the function space (space of functions represented by MPS) is guaranteed without any extra assumptions of the data set. We then consider the settings of (supervised) Regression with Mean Square Error (RMSE) and (unsupervised) Born Machines (BM) and analyze their dynamics in the infinite bond dimensional limit. The ordinary differential equations (ODEs) which describe the dynamics of the responses of MPS in the RMSE and BM are derived and solved in the closed-form. For the Regression, we consider Mercer Kernels (Gaussian Kernels) and find that the evolution of the mean of the responses of MPS follows the largest eigenvalue of the NTK. Due to the orthogonality of the kernel functions in BM, the evolution of different modes (samples) decouples and the "characteristic time" of convergence in training is obtained.
Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets
Authors: Shao-Bo Lin, Yao Wang, Ding-Xuan Zhou
Date: 2021-11-28 00:00:00

In this paper, we study the generalization performance of global minima for implementing empirical risk minimization (ERM) on over-parameterized deep ReLU nets. Using a novel deepening scheme for deep ReLU nets, we rigorously prove that there exist perfect global minima achieving almost optimal generalization error bounds for numerous types of data under mild conditions. Since over-parameterization is crucial to guarantee that the global minima of ERM on deep ReLU nets can be realized by the widely used stochastic gradient descent (SGD) algorithm, our results indeed fill a gap between optimization and generalization.
Learning Wildfire Model from Incomplete State Observations
Authors: Alissa Chavalithumrong, Hyung-Jin Yoon, Petros Voulgaris
Date: 2021-11-28 00:00:00
Tasks: Management, model, Prediction, Time Series, Time Series Analysis

As wildfires are expected to become more frequent and severe, improved prediction models are vital to mitigating risk and allocating resources. With remote sensing data, valuable spatiotemporal statistical models can be created and used for resource management practices. In this paper, we create a dynamic model for future wildfire predictions of five locations within the western United States through a deep neural network via historical burned area and climate data. The proposed model has distinct features that address the characteristic need in prediction evaluations, including dynamic online estimation and time-series modeling. Between locations, local fire event triggers are not isolated, and there are confounding factors when local data is analyzed due to incomplete state observations. When compared to existing approaches that do not account for incomplete state observation within wildfire time-series data, on average, we are able to achieve higher prediction performances.
FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding
Authors: Bill Tuck Weng Pung, Alvin Chan
Date: 2021-11-28 00:00:00
Tasks: Language Modeling, Language Modelling, Machine Translation, Natural Language Inference, Sentiment Analysis

Inducing latent tree structures from sequential data is an emerging trend in the NLP research landscape today, largely popularized by recent methods such as Gumbel LSTM and Ordered Neurons (ON-LSTM). This paper proposes FASTTREES, a new general purpose neural module for fast sequence encoding. Unlike most previous works that consider recurrence to be necessary for tree induction, our work explores the notion of parallel tree induction, i.e., imbuing our model with hierarchical inductive biases in a parallelizable, non-autoregressive fashion. To this end, our proposed FASTTREES achieves competitive or superior performance to ON-LSTM on four well-established sequence modeling tasks, i.e., language modeling, logical inference, sentiment analysis and natural language inference. Moreover, we show that the FASTTREES module can be applied to enhance Transformer models, achieving performance gains on three sequence transduction tasks (machine translation, subject-verb agreement and mathematical language understanding), paving the way for modular tree induction modules. Overall, we outperform existing state-of-the-art models on logical inference tasks by +4% and mathematical language understanding by +8%.
Federated Gaussian Process: Convergence, Automatic Personalization and Multi-fidelity Modeling
Authors: Xubo Yue, Raed Al Kontar
Date: 2021-11-28 00:00:00
Tasks: Privacy Preserving

In this paper, we propose \texttt{FGPR}: a Federated Gaussian process ($\mathcal{GP}$) regression framework that uses an averaging strategy for model aggregation and stochastic gradient descent for local client computations. Notably, the resulting global model excels in personalization as \texttt{FGPR} jointly learns a global $\mathcal{GP}$ prior across all clients. The predictive posterior then is obtained by exploiting this prior and conditioning on local data which encodes personalized features from a specific client. Theoretically, we show that \texttt{FGPR} converges to a critical point of the full log-likelihood function, subject to statistical error. Through extensive case studies we show that \texttt{FGPR} excels in a wide range of applications and is a promising approach for privacy-preserving multi-fidelity data modeling.
An Entropy Weighted Nonnegative Matrix Factorization Algorithm for Feature Representation
Authors: Jiao Wei, Can Tong, Bingxue Wu, Qiang He, Shouliang Qi, YuDong Yao, Yueyang Teng
Date: 2021-11-27 00:00:00
Tasks: Attribute

Nonnegative matrix factorization (NMF) has been widely used to learn low-dimensional representations of data. However, NMF pays the same attention to all attributes of a data point, which inevitably leads to inaccurate representation. For example, in a human-face data set, if an image contains a hat on the head, the hat should be removed or the importance of its corresponding attributes should be decreased during matrix factorizing. This paper proposes a new type of NMF called entropy weighted NMF (EWNMF), which uses an optimizable weight for each attribute of each data point to emphasize their importance. This process is achieved by adding an entropy regularizer to the cost function and then using the Lagrange multiplier method to solve the problem. Experimental results with several data sets demonstrate the feasibility and effectiveness of the proposed method. We make our code available at https://github.com/Poisson-EM/Entropy-weighted-NMF.
Answer Generation for Questions With Multiple Information Sources in E-Commerce
Authors: Anand A. Rajasekar, Nikesh Garera
Date: 2021-11-27 00:00:00
Tasks: Answer Generation, Question Answering

Automatic question answering is an important yet challenging task in E-commerce given the millions of questions posted by users about the product that they are interested in purchasing. Hence, there is a great demand for automatic answer generation systems that provide quick responses using related information about the product. There are three sources of knowledge available for answering a user posted query, they are reviews, duplicate or similar questions, and specifications. Effectively utilizing these information sources will greatly aid us in answering complex questions. However, there are two main challenges present in exploiting these sources: (i) The presence of irrelevant information and (ii) the presence of ambiguity of sentiment present in reviews and similar questions. Through this work we propose a novel pipeline (MSQAP) that utilizes the rich information present in the aforementioned sources by separately performing relevancy and ambiguity prediction before generating a response. Experimental results show that our relevancy prediction model (BERT-QA) outperforms all other variants and has an improvement of 12.36% in F1 score compared to the BERT-base baseline. Our generation model (T5-QA) outperforms the baselines in all content preservation metrics such as BLEU, ROUGE and has an average improvement of 35.02% in ROUGE and 198.75% in BLEU compared to the highest performing baseline (HSSC-q). Human evaluation of our pipeline shows us that our method has an overall improvement in accuracy of 30.7% over the generation model (T5-QA), resulting in our full pipeline-based approach (MSQAP) providing more accurate answers. To the best of our knowledge, this is the first work in the e-commerce domain that automatically generates natural language answers combining the information present in diverse sources such as specifications, similar questions, and reviews data.
Factor-augmented tree ensembles
Authors: Filippo Pellegrino
Date: 2021-11-27 00:00:00
Tasks: Ensemble Learning, regression, Time Series, Time Series Analysis, Time Series Regression

This manuscript proposes to extend the information set of time-series regression trees with latent stationary factors extracted via state-space methods. In doing so, this approach generalises time-series regression trees on two dimensions. First, it allows to handle predictors that exhibit measurement error, non-stationary trends, seasonality and/or irregularities such as missing observations. Second, it gives a transparent way for using domain-specific theory to inform time-series regression trees. Empirically, ensembles of these factor-augmented trees provide a reliable approach for macro-finance problems. This article highlights it focussing on the lead-lag effect between equity volatility and the business cycle in the United States.
Multi-modality fusion using canonical correlation analysis methods: Application in breast cancer survival prediction from histology and genomics
Authors: Vaishnavi Subramanian, Tanveer Syeda-Mahmood, Minh N. Do
Date: 2021-11-27 00:00:00
Tasks: Prediction, Survival Prediction

The availability of multi-modality datasets provides a unique opportunity to characterize the same object of interest using multiple viewpoints more comprehensively. In this work, we investigate the use of canonical correlation analysis (CCA) and penalized variants of CCA (pCCA) for the fusion of two modalities. We study a simple graphical model for the generation of two-modality data. We analytically show that, with known model parameters, posterior mean estimators that jointly use both modalities outperform arbitrary linear mixing of single modality posterior estimators in latent variable prediction. Penalized extensions of CCA (pCCA) that incorporate domain knowledge can discover correlations with high-dimensional, low-sample data, whereas traditional CCA is inapplicable. To facilitate the generation of multi-dimensional embeddings with pCCA, we propose two matrix deflation schemes that enforce desirable properties exhibited by CCA. We propose a two-stage prediction pipeline using pCCA embeddings generated with deflation for latent variable prediction by combining all the above. On simulated data, our proposed model drastically reduces the mean-squared error in latent variable prediction. When applied to publicly available histopathology data and RNA-sequencing data from The Cancer Genome Atlas (TCGA) breast cancer patients, our model can outperform principal components analysis (PCA) embeddings of the same dimension in survival prediction.
Forecasting Daily COVID-19 Related Calls in VA Health Care System: Predictive Model Development
Authors: Weipeng Zhou, Ryan J. Laundry, Paul L. Hebert, Gang Luo
Date: 2021-11-27 00:00:00
Tasks: feature selection

Background: COVID-19 has become a challenge worldwide and properly planning of medical resources is the key to combating COVID-19. In the US Veteran Affairs Health Care System (VA), many of the enrollees are susceptible to COVID-19. Predicting the COVID-19 to allocate medical resources promptly becomes a critical issue. When the VA enrollees have COVID-19 symptoms, it is recommended that their first step should be to call the VA Call Center. For confirmed COVID-19 patients, the median time from the first symptom to hospital admission was seven days. By predicting the number of COVID-19 related calls, we could predict imminent surges in healthcare use and plan medical resources ahead. Objective: The study aims to develop a method to forecast the daily number of COVID-19 related calls for each of the 110 VA medical centers. Methods: In the proposed method, we pre-trained a model using a cluster of medical centers and fine-tuned it for individual medical centers. At the cluster level, we performed feature selection to select significant features and automatic hyper-parameter search to select optimal hyper-parameter value combinations for the model. Conclusions: This study proposed an accurate method to forecast the daily number of COVID-19 related calls for VA medical centers. The proposed method was able to overcome modeling challenges by grouping similar medical centers into clusters to enlarge the dataset for training models, and using hyper-parameter search to automatically find optimal hyper-parameter value combinations for models. With the proposed method, surges in health care can be predicted ahead. This allows health care practitioners to better plan medical resources and combat COVID-19.
Benchmarking Accuracy and Generalizability of Four Graph Neural Networks Using Large In Vitro ADME Datasets from Different Chemical Spaces
Authors: Fabio Broccatelli, Richard Trager, Michael Reutlinger, George Karypis, Mufei Li
Date: 2021-11-27 00:00:00
Tasks: Benchmarking, Graph Attention, Graph Neural Network

In this work, we benchmark a variety of single- and multi-task graph neural network (GNN) models against lower-bar and higher-bar traditional machine learning approaches employing human engineered molecular features. We consider four GNN variants -- Graph Convolutional Network (GCN), Graph Attention Network (GAT), Message Passing Neural Network (MPNN), and Attentive Fingerprint (AttentiveFP). So far deep learning models have been primarily benchmarked using lower-bar traditional models solely based on fingerprints, while more realistic benchmarks employing fingerprints, whole-molecule descriptors and predictions from other related endpoints (e.g., LogD7.4) appear to be scarce for industrial ADME datasets. In addition to time-split test sets based on Genentech data, this study benefits from the availability of measurements from an external chemical space (Roche data). We identify GAT as a promising approach to implementing deep learning models. While all GNN models significantly outperform lower-bar benchmark traditional models solely based on fingerprints, only GATs seem to offer a small but consistent improvement over higher-bar benchmark traditional models. Finally, the accuracy of in vitro assays from different laboratories predicting the same experimental endpoints appears to be comparable with the accuracy of GAT single-task models, suggesting that most of the observed error from the models is a function of the experimental error propagation.
Leveraging Unsupervised Learning to Summarize APIs Discussed in Stack Overflow
Authors: AmirHossein Naghshzan, Latifa Guerrouj, Olga Baysal
Date: 2021-11-27 00:00:00
Tasks: Code Summarization, Source Code Summarization

Automated source code summarization is a task that generates summarized information about the purpose, usage, and--or implementation of methods and classes to support understanding of these code entities. Multiple approaches and techniques have been proposed for supervised and unsupervised learning in code summarization, however, they were mostly focused on generating a summary for a piece of code. In addition, very few works have leveraged unofficial documentation. This paper proposes an automatic and novel approach for summarizing Android API methods discussed in Stack Overflow that we consider as unofficial documentation in this research. Our approach takes the API method's name as an input and generates a natural language summary based on Stack Overflow discussions of that API method. We have conducted a survey that involves 16 Android developers to evaluate the quality of our automatically generated summaries and compare them with the official Android documentation. Our results demonstrate that while developers find the official documentation more useful in general, the generated summaries are also competitive, in particular for offering implementation details, and can be used as a complementary source for guiding developers in software development and maintenance tasks.
Understanding Anharmonic Effects on Hydrogen Desorption Characteristics of Mg$_n$H$_{2n}$ Nanoclusters by ab initio trained Deep Neural Network
Authors: Andrea Pedrielli, Paolo E. Trevisanutto, Lorenzo Monacelli, Giovanni Garberoglio, Nicola M. Pugno, Simone Taioli
Date: 2021-11-27 00:00:00

Magnesium hydride (MgH$_2$) has been widely studied for effective hydrogen storage. However, its bulk desorption temperature (553 K) is deemed too high for practical applications. Besides doping, a strategy to decrease such reaction energy for releasing hydrogen is the use of MgH$_2$-based nanoparticles (NPs). Here, we investigate first the thermodynamic properties of Mg$_n$H$_{2n}$ NPs ($n<10$) from first-principles, in particular by assessing the anharmonic effects on the enthalpy, entropy and thermal expansion by means of the Stochastic Self Consistent Harmonic Approximation (SSCHA). The latter method goes beyond previous approaches, typically based on molecular mechanics and the quasi-harmonic approximation, allowing the ab initio calculation of the fully-anharmonic free energy. We find an almost linear dependence on temperature of the interatomic bond lengths - with a relative variation of few percent over 300K -, alongside with a bond distance decrease of the Mg-H bonds. In order to increase the size of NPs toward experiments of hydrogen desorption from MgH$_2$ we devise a computationally effective Machine Learning model trained to accurately determine the forces and total energies (i.e. the potential energy surfaces), integrating the latter with the SSCHA model to fully include the anharmonic effects. We find a significative decrease of the H-desorption temperature for sub-nanometric clusters Mg$_n$H$_{2n}$ with $n \leq 10$, with a non-negligible, although little effect due to anharmonicities (up to 10%).
A Recommender System-Inspired Cloud Data Filling Scheme for Satellite-based Coastal Observation
Authors: Ruo-Qian Wang
Date: 2021-11-27 00:00:00
Tasks: Earth Observation, Imputation, Recommendation Systems

Filling missing data in cloud-covered areas of satellite imaging is an important task to improve data quantity and quality for enhanced earth observation. Traditional cloud filling studies focused on continuous numerical data such as temperature and cyanobacterial concentration in the open ocean. Cloud data filling issues in coastal imaging is far less studied because of the complex landscape. Inspired by the success of data imputation methods in recommender systems that are designed for online shopping, the present study explored their application to satellite cloud data filling tasks. A numerical experiment was designed and conducted for a LandSat dataset with a range of synthetic cloud covers to examine the performance of different data filling schemes. The recommender system-inspired matrix factorization algorithm called Funk-SVD showed superior performance in computational accuracy and efficiency for the task of recovering landscape types in a complex coastal area than the traditional data filling scheme of DINEOF (Data Interpolating Empirical Orthogonal Functions) and the deep learning method of Datawig. The new method achieved the best filling accuracy and reached a speed comparable to DINEOF and much faster than deep learning. A theoretical framework was created to analyze the error propagation in DINEOF and found the algorithm needs to be modified to converge to the ground truth. The present study showed that Funk-SVD has great potential to enhance cloud data filling performance and connects the fields of recommender systems and cloud filling to promote the improvement and sharing of useful algorithms.
Resource-Aware Asynchronous Online Federated Learning for Nonlinear Regression
Authors: Francois Gauthier, Vinay Chakravarthi Gogineni, Stefan Werner, Yih-Fang Huang, Anthony Kuh
Date: 2021-11-27 00:00:00
Tasks: Federated Learning, regression

Many assumptions in the federated learning literature present a best-case scenario that can not be satisfied in most real-world applications. An asynchronous setting reflects the realistic environment in which federated learning methods must be able to operate reliably. Besides varying amounts of non-IID data at participants, the asynchronous setting models heterogeneous client participation due to available computational power and battery constraints and also accounts for delayed communications between clients and the server. To reduce the communication overhead associated with asynchronous online federated learning (ASO-Fed), we use the principles of partial-sharing-based communication. In this manner, we reduce the communication load of the participants and, therefore, render participation in the learning task more accessible. We prove the convergence of the proposed ASO-Fed and provide simulations to analyze its behavior further. The simulations reveal that, in the asynchronous setting, it is possible to achieve the same convergence as the federated stochastic gradient (Online-FedSGD) while reducing the communication tenfold.
Transformed K-means Clustering
Authors: Anurag Goel, Angshul Majumdar
Date: 2021-11-27 00:00:00
Tasks: Clustering

In this work we propose a clustering framework based on the paradigm of transform learning. In simple terms the representation from transform learning is used for K-means clustering; however, the problem is not solved in such a na\"ive piecemeal fashion. The K-means clustering loss is embedded into the transform learning framework and the joint problem is solved using the alternating direction method of multipliers. Results on document clustering show that our proposed approach improves over the state-of-the-art.
Fast and Informative Model Selection using Learning Curve Cross-Validation
Authors: Felix Mohr, Jan N. van Rijn
Date: 2021-11-27 00:00:00
Tasks: AutoML, Model Selection

Common cross-validation (CV) methods like k-fold cross-validation or Monte-Carlo cross-validation estimate the predictive performance of a learner by repeatedly training it on a large portion of the given data and testing on the remaining data. These techniques have two major drawbacks. First, they can be unnecessarily slow on large datasets. Second, beyond an estimation of the final performance, they give almost no insights into the learning process of the validated algorithm. In this paper, we present a new approach for validation based on learning curves (LCCV). Instead of creating train-test splits with a large portion of training data, LCCV iteratively increases the number of instances used for training. In the context of model selection, it discards models that are very unlikely to become competitive. We run a large scale experiment on the 67 datasets from the AutoML benchmark and empirically show that in over 90% of the cases using LCCV leads to similar performance (at most 1.5% difference) as using 5/10-fold CV. However, it yields substantial runtime reductions of over 20% on average. Additionally, it provides important insights, which for example allow assessing the benefits of acquiring more data. These results are orthogonal to other advances in the field of AutoML.