pwc-archive/papers-with-abstracts huggingface dataset:
Multi-Centroid Hyperdimensional Computing Approach for Epileptic Seizure Detection
Authors: Una Pale, Tomas Teijeiro, David Atienza
Date: 2021-11-16 00:00:00
Tasks: EEG, Electroencephalogram (EEG), Seizure Detection

Long-term monitoring of patients with epilepsy presents a challenging problem from the engineering perspective of real-time detection and wearable devices design. It requires new solutions that allow continuous unobstructed monitoring and reliable detection and prediction of seizures. A high variability in the electroencephalogram (EEG) patterns exists among people, brain states, and time instances during seizures, but also during non-seizure periods. This makes epileptic seizure detection very challenging, especially if data is grouped under only seizure and non-seizure labels. Hyperdimensional (HD) computing, a novel machine learning approach, comes in as a promising tool. However, it has certain limitations when the data shows a high intra-class variability. Therefore, in this work, we propose a novel semi-supervised learning approach based on a multi-centroid HD computing. The multi-centroid approach allows to have several prototype vectors representing seizure and non-seizure states, which leads to significantly improved performance when compared to a simple 2-class HD model. Further, real-life data imbalance poses an additional challenge and the performance reported on balanced subsets of data is likely to be overestimated. Thus, we test our multi-centroid approach with three different dataset balancing scenarios, showing that performance improvement is higher for the less balanced dataset. More specifically, up to 14% improvement is achieved on an unbalanced test set with 10 times more non-seizure than seizure data. At the same time, the total number of sub-classes is not significantly increased compared to the balanced dataset. Thus, the proposed multi-centroid approach can be an important element in achieving a high performance of epilepsy detection with real-life data balance or during online learning, where seizures are infrequent.
Thoughts on the Consistency between Ricci Flow and Neural Network Behavior
Authors: Jun Chen, Tianxin Huang, Wenzhou Chen, Yong liu
Date: 2021-11-16 00:00:00

The Ricci flow is a partial differential equation for evolving the metric in a Riemannian manifold to make it more regular. On the other hand, neural networks seem to have similar geometric behavior for specific tasks. In this paper, we construct the linearly nearly Euclidean manifold as a background to observe the evolution of Ricci flow and the training of neural networks. Under the Ricci-DeTurck flow, we prove the dynamical stability and convergence of the linearly nearly Euclidean metric for an $L^2$-Norm perturbation. In practice, from the information geometry and mirror descent points of view, we give the steepest descent gradient flow for neural networks on the linearly nearly Euclidean manifold. During the training process of the neural network, we observe that its metric will also regularly converge to the linearly nearly Euclidean metric, which is consistent with the convergent behavior of linearly nearly Euclidean metrics under the Ricci-DeTurck flow.
CLARA: A Constrained Reinforcement Learning Based Resource Allocation Framework for Network Slicing
Authors: Yongshuai Liu, Jiaxin Ding, Zhi-Li Zhang, Xin Liu
Date: 2021-11-16 00:00:00
Tasks: Management, reinforcement-learning, Reinforcement Learning (RL)

As mobile networks proliferate, we are experiencing a strong diversification of services, which requires greater flexibility from the existing network. Network slicing is proposed as a promising solution for resource utilization in 5G and future networks to address this dire need. In network slicing, dynamic resource orchestration and network slice management are crucial for maximizing resource utilization. Unfortunately, this process is too complex for traditional approaches to be effective due to a lack of accurate models and dynamic hidden structures. We formulate the problem as a Constrained Markov Decision Process (CMDP) without knowing models and hidden structures. Additionally, we propose to solve the problem using CLARA, a Constrained reinforcement LeArning based Resource Allocation algorithm. In particular, we analyze cumulative and instantaneous constraints using adaptive interior-point policy optimization and projection layer, respectively. Evaluations show that CLARA clearly outperforms baselines in resource allocation with service demand guarantees.
On-Demand Unlabeled Personalized Federated Learning
Authors: Ohad Amosy, Gal Eyal, Gal Chechik
Date: 2021-11-16 00:00:00
Tasks: Domain Adaptation, Federated Learning, Multi-Task Learning, Personalized Federated Learning

In Federated Learning (FL), multiple clients collaborate to learn a shared model through a central server while keeping data decentralized. Personalized Federated Learning (PFL) further extends FL by learning a personalized model per client. In both FL and PFL, all clients participate in the training process and their labeled data are used for training. However, in reality, novel clients may wish to join a prediction service after it has been deployed, obtaining predictions for their own \textbf{unlabeled} data. Here, we introduce a new learning setup, On-Demand Unlabeled PFL (OD-PFL), where a system trained on a set of clients, needs to be later applied to novel unlabeled clients at inference time. We propose a novel approach to this problem, ODPFL-HN, which learns to produce a new model for the late-to-the-party client. Specifically, we train an encoder network that learns a representation for a client given its unlabeled data. That client representation is fed to a hypernetwork that generates a personalized model for that client. Evaluated on five benchmark datasets, we find that ODPFL-HN generalizes better than the current FL and PFL methods, especially when the novel client has a large shift from training clients. We also analyzed the generalization error for novel clients, and showed analytically and experimentally how novel clients can apply differential privacy.
Mathematical Models for Local Sensing Hashes
Authors: Li Wang, Lilon Wangner
Date: 2021-11-16 00:00:00
Tasks: Clustering, Outlier Detection

As data volumes continue to grow, searches in data are becoming increasingly time-consuming. Classical index structures for neighbor search are no longer sustainable due to the "curse of dimensionality". Instead, approximated index structures offer a good opportunity to significantly accelerate the neighbor search for clustering and outlier detection and to have the lowest possible error rate in the results of the algorithms. Local sensing hashes is one of those. We indicate directions to mathematically model the properties of it.
Reshaping Smart Energy Transition: An analysis of human-building interactions in Qatar Using Machine Learning Techniques
Authors: Rateb Jabbar, Esmat Zaidan, Ahmed Ben Said, Ali Ghofrani
Date: 2021-11-16 00:00:00
Tasks: Feature Importance

Policy Planning have the potential to contribute to the strategic development and economic diversification of developing countries even without considerable structural changes. In this study, we analyzed a set of human-oriented dimensions aimed at improving energy policies related to the building sector in Qatar. Considering the high percentage of expatriate and migrant communities with different financial and cultural backgrounds and behavioral patterns compared with local communities in the GCC Union, it is required to investigate human dimensions to propose adequate energy policies. This study explored the correlations of socioeconomic, behavioral, and demographic dimensions to determine the main factors behind discrepancies in energy use, responsibilities, motivations, habits, and overall well-being. The sample included 2,200 people in Qatar, and it was clustered into two consumer categories: high and low. In particular, the study focused on exploring human indoor comfort perception dependencies with building features. Financial drivers, such as demand programs and energy subsidies, were explored in relation to behavioral patterns. Subsequently, the data analysis resulted in implications for energy policies regarding interventions, social well-being, and awareness. Machine learning methods were used to perform a feature importance analysis to determine the main factors of human behavior. The findings of this study demonstrated how human factors impact comfort perception in residential and work environments, norms, habits, self-responsibility, consequence awareness, and consumption. The study has important implications for developing targeted strategies aimed at improving the efficacy of energy policies and sustainability performance indicators.
Bayesian Optimization for Cascade-type Multi-stage Processes
Authors: Shunya Kusakawa, Shion Takeno, Yu Inatsu, Kentaro Kutsukake, Shogo Iwazaki, Takashi Nakano, Toru Ujihara, Masayuki Karasuyama, Ichiro Takeuchi
Date: 2021-11-16 00:00:00
Tasks: Bayesian Optimization, Decision Making, Vocal Bursts Type Prediction

Complex processes in science and engineering are often formulated as multistage decision-making problems. In this paper, we consider a type of multistage decision-making process called a cascade process. A cascade process is a multistage process in which the output of one stage is used as an input for the subsequent stage. When the cost of each stage is expensive, it is difficult to search for the optimal controllable parameters for each stage exhaustively. To address this problem, we formulate the optimization of the cascade process as an extension of the Bayesian optimization framework and propose two types of acquisition functions based on credible intervals and expected improvement. We investigate the theoretical properties of the proposed acquisition functions and demonstrate their effectiveness through numerical experiments. In addition, we consider an extension called suspension setting in which we are allowed to suspend the cascade process at the middle of the multistage decision-making process that often arises in practical problems. We apply the proposed method in a test problem involving a solar cell simulator, which was the motivation for this study.
Learning with convolution and pooling operations in kernel methods
Authors: Theodor Misiakiewicz, Song Mei
Date: 2021-11-16 00:00:00
Tasks: image-classification, Image Classification

Recent empirical work has shown that hierarchical convolutional kernels inspired by convolutional neural networks (CNNs) significantly improve the performance of kernel methods in image classification tasks. A widely accepted explanation for their success is that these architectures encode hypothesis classes that are suitable for natural images. However, understanding the precise interplay between approximation and generalization in convolutional architectures remains a challenge. In this paper, we consider the stylized setting of covariates (image pixels) uniformly distributed on the hypercube, and characterize exactly the RKHS of kernels composed of single layers of convolution, pooling, and downsampling operations. We use this characterization to compute sharp asymptotics of the generalization error for any given function in high-dimension. In particular, we quantify the gain in sample complexity brought by enforcing locality with the convolution operation and approximate translation invariance with average pooling. Notably, these results provide a precise description of how convolution and pooling operations trade off approximation with generalization power in one layer convolutional kernels.
Machine Learning-Based Assessment of Energy Behavior of RC Shear Walls
Authors: Berkay Topaloglu, Gulsen Taskin Kaya, Fatih Sutcu, Zeynep Tuna Deger
Date: 2021-11-16 00:00:00
Tasks: BIG-bench Machine Learning, feature selection, GPR, Philosophy

Current seismic design codes primarily rely on the strength and displacement capacity of structural members and do not account for the influence of the ground motion duration or the hysteretic behavior characteristics. The energy-based approach serves as a supplemental index to response quantities and includes the effect of repeated loads in seismic performance. The design philosophy suggests that the seismic demands are met by the energy dissipation capacity of the structural members. Therefore, the energy dissipation behavior of the structural members should be well understood to achieve an effective energy-based design approach. This study focuses on the energy dissipation capacity of reinforced concrete (RC) shear walls that are widely used in high seismic regions as they provide significant stiffness and strength to resist lateral forces. A machine learning (Gaussian Process Regression (GPR))-based predictive model for energy dissipation capacity of shear walls is developed as a function of wall design parameters. Eighteen design parameters are shown to influence energy dissipation, whereas the most important ones are determined by applying sequential backward elimination and by using feature selection methods to reduce the complexity of the predictive model. The ability of the proposed model to make robust and accurate predictions is validated based on novel data with a prediction accuracy (the ratio of predicted/actual values) of around 1.00 and a coefficient of determination (R2) of 0.93. The outcomes of this study are believed to contribute to the energy-based approach by (i) defining the most influential wall properties on the seismic energy dissipation capacity of shear walls and (ii) providing predictive models that can enable comparisons of different wall design configurations to achieve higher energy dissipation capacity.
Wyner-Ziv Gradient Compression for Federated Learning
Authors: Kai Liang, Huiru Zhong, Haoning Chen, Youlong Wu
Date: 2021-11-16 00:00:00
Tasks: Federated Learning, Quantization

Due to limited communication resources at the client and a massive number of model parameters, large-scale distributed learning tasks suffer from communication bottleneck. Gradient compression is an effective method to reduce communication load by transmitting compressed gradients. Motivated by the fact that in the scenario of stochastic gradients descent, gradients between adjacent rounds may have a high correlation since they wish to learn the same model, this paper proposes a practical gradient compression scheme for federated learning, which uses historical gradients to compress gradients and is based on Wyner-Ziv coding but without any probabilistic assumption. We also implement our gradient quantization method on the real dataset, and the performance of our method is better than the previous schemes.
Deep Distilling: automated code generation using explainable deep learning
Authors: Paul J. Blazek, Kesavan Venkatesh, Milo M. Lin
Date: 2021-11-16 00:00:00
Tasks: Code Generation, Common Sense Reasoning, Deep Learning

Human reasoning can distill principles from observed patterns and generalize them to explain and solve novel problems. The most powerful artificial intelligence systems lack explainability and symbolic reasoning ability, and have therefore not achieved supremacy in domains requiring human understanding, such as science or common sense reasoning. Here we introduce deep distilling, a machine learning method that learns patterns from data using explainable deep learning and then condenses it into concise, executable computer code. The code, which can contain loops, nested logical statements, and useful intermediate variables, is equivalent to the neural network but is generally orders of magnitude more compact and human-comprehensible. On a diverse set of problems involving arithmetic, computer vision, and optimization, we show that deep distilling generates concise code that generalizes out-of-distribution to solve problems orders-of-magnitude larger and more complex than the training data. For problems with a known ground-truth rule set, deep distilling discovers the rule set exactly with scalable guarantees. For problems that are ambiguous or computationally intractable, the distilled rules are similar to existing human-derived algorithms and perform at par or better. Our approach demonstrates that unassisted machine intelligence can build generalizable and intuitive rules explaining patterns in large datasets that would otherwise overwhelm human reasoning.
Pre-training Graph Neural Network for Cross Domain Recommendation
Authors: Chen Wang, Yueqing Liang, Zhiwei Liu, Tao Zhang, Philip S. Yu
Date: 2021-11-16 00:00:00
Tasks: Graph Neural Network, Graph Representation Learning, Recommendation Systems, Representation Learning

A recommender system predicts users' potential interests in items, where the core is to learn user/item embeddings. Nevertheless, it suffers from the data-sparsity issue, which the cross-domain recommendation can alleviate. However, most prior works either jointly learn the source domain and target domain models, or require side-features. However, jointly training and side features would affect the prediction on the target domain as the learned embedding is dominated by the source domain containing bias information. Inspired by the contemporary arts in pre-training from graph representation learning, we propose a pre-training and fine-tuning diagram for cross-domain recommendation. We devise a novel Pre-training Graph Neural Network for Cross-Domain Recommendation (PCRec), which adopts the contrastive self-supervised pre-training of a graph encoder. Then, we transfer the pre-trained graph encoder to initialize the node embeddings on the target domain, which benefits the fine-tuning of the single domain recommender system on the target domain. The experimental results demonstrate the superiority of PCRec. Detailed analyses verify the superiority of PCRec in transferring information while avoiding biases from source domains.
A Unified and Fast Interpretable Model for Predictive Analytics
Authors: Yuanyuan Jiang, Rui Ding, Tianchi Qiao, Yunan Zhu, Shi Han, Dongmei Zhang
Date: 2021-11-16 00:00:00
Tasks: Decision Making

Predictive analytics aims to build machine learning models to predict behavior patterns and use predictions to guide decision-making. Predictive analytics is human involved, thus the machine learning model is preferred to be interpretable. In literature, Generalized Additive Model (GAM) is a standard for interpretability. However, due to the one-to-many and many-to-one phenomena which appear commonly in real-world scenarios, existing GAMs have limitations to serve predictive analytics in terms of both accuracy and training efficiency. In this paper, we propose FXAM (Fast and eXplainable Additive Model), a unified and fast interpretable model for predictive analytics. FXAM extends GAM's modeling capability with a unified additive model for numerical, categorical, and temporal features. FXAM conducts a novel training procedure called Three-Stage Iteration (TSI). TSI corresponds to learning over numerical, categorical, and temporal features respectively. Each stage learns a local optimum by fixing the parameters of other stages. We design joint learning over categorical features and partial learning over temporal features to achieve high accuracy and training efficiency. We prove that TSI is guaranteed to converge to the global optimum. We further propose a set of optimization techniques to speed up FXAM's training algorithm to meet the needs of interactive analysis. Thorough evaluations conducted on diverse data sets verify that FXAM significantly outperforms existing GAMs in terms of training speed, and modeling categorical and temporal features. In terms of interpretability, we compare FXAM with the typical post-hoc approach XGBoost+SHAP on two real-world scenarios, which shows the superiority of FXAM's inherent interpretability for predictive analytics.
Assessing Deep Neural Networks as Probability Estimators
Authors: Yu Pan, Kwo-Sen Kuo, Michael L. Rilee, Hongfeng Yu
Date: 2021-11-16 00:00:00
Tasks: Classification

Deep Neural Networks (DNNs) have performed admirably in classification tasks. However, the characterization of their classification uncertainties, required for certain applications, has been lacking. In this work, we investigate the issue by assessing DNNs' ability to estimate conditional probabilities and propose a framework for systematic uncertainty characterization. Denoting the input sample as x and the category as y, the classification task of assigning a category y to a given input x can be reduced to the task of estimating the conditional probabilities p(y|x), as approximated by the DNN at its last layer using the softmax function. Since softmax yields a vector whose elements all fall in the interval (0, 1) and sum to 1, it suggests a probabilistic interpretation to the DNN's outcome. Using synthetic and real-world datasets, we look into the impact of various factors, e.g., probability density f(x) and inter-categorical sparsity, on the precision of DNNs' estimations of p(y|x), and find that the likelihood probability density and the inter-categorical sparsity have greater impacts than the prior probability to DNNs' classification uncertainty.
Covariate Shift in High-Dimensional Random Feature Regression
Authors: Nilesh Tripuraneni, Ben Adlam, Jeffrey Pennington
Date: 2021-11-16 00:00:00
Tasks: BIG-bench Machine Learning, Out-of-Distribution Generalization, regression, Vocal Bursts Intensity Prediction

A significant obstacle in the development of robust machine learning models is covariate shift, a form of distribution shift that occurs when the input distributions of the training and test sets differ while the conditional label distributions remain the same. Despite the prevalence of covariate shift in real-world applications, a theoretical understanding in the context of modern machine learning has remained lacking. In this work, we examine the exact high-dimensional asymptotics of random feature regression under covariate shift and present a precise characterization of the limiting test error, bias, and variance in this setting. Our results motivate a natural partial order over covariate shifts that provides a sufficient condition for determining when the shift will harm (or even help) test performance. We find that overparameterized models exhibit enhanced robustness to covariate shift, providing one of the first theoretical explanations for this intriguing phenomenon. Additionally, our analysis reveals an exact linear relationship between in-distribution and out-of-distribution generalization performance, offering an explanation for this surprising recent empirical observation.
Selective Ensembles for Consistent Predictions
Authors: Emily Black, Klas Leino, Matt Fredrikson
Date: 2021-11-16 00:00:00
Proceeding: ICLR 2022 4
Tasks: Medical Diagnosis

Recent work has shown that models trained to the same objective, and which achieve similar measures of accuracy on consistent test data, may nonetheless behave very differently on individual predictions. This inconsistency is undesirable in high-stakes contexts, such as medical diagnosis and finance. We show that this inconsistent behavior extends beyond predictions to feature attributions, which may likewise have negative implications for the intelligibility of a model, and one's ability to find recourse for subjects. We then introduce selective ensembles to mitigate such inconsistencies by applying hypothesis testing to the predictions of a set of models trained using randomly-selected starting conditions; importantly, selective ensembles can abstain in cases where a consistent outcome cannot be achieved up to a specified confidence level. We prove that that prediction disagreement between selective ensembles is bounded, and empirically demonstrate that selective ensembles achieve consistent predictions and feature attributions while maintaining low abstention rates. On several benchmark datasets, selective ensembles reach zero inconsistently predicted points, with abstention rates as low 1.5%.