pwc-archive/papers-with-abstracts huggingface dataset:
Supervised Robustness-preserving Data-free Neural Network Pruning
Authors: Mark Huasong Meng, Guangdong Bai, Sin Gee Teo, Jin Song Dong
Date: 2022-04-02 00:00:00
Tasks: Network Pruning, Stochastic Optimization

When deploying pre-trained neural network models in real-world applications, model consumers often encounter resource-constraint platforms such as mobile and smart devices. They typically use the pruning technique to reduce the size and complexity of the model, generating a lighter one with less resource consumption. Nonetheless, most existing pruning methods are proposed with the premise that the model after being pruned has a chance to be fine-tuned or even retrained based on the original training data. This may be unrealistic in practice, as the data controllers are often reluctant to provide their model consumers with the original data. In this work, we study the neural network pruning in the data-free context, aiming to yield lightweight models that are not only accurate in prediction but also robust against undesired inputs in open-world deployments. Considering the absence of the fine-tuning and retraining that can fix the mis-pruned units, we replace the traditional aggressive one-shot strategy with a conservative one that treats the pruning as a progressive process. We propose a pruning method based on stochastic optimization that uses robustness-related metrics to guide the pruning process. Our method is implemented as a Python program and evaluated with a series of experiments on diverse neural network models. The experimental results show that it significantly outperforms existing one-shot data-free pruning approaches in terms of robustness preservation and accuracy.
AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio
Authors: Jun Lu
Date: 2022-04-02 00:00:00
Tasks: Stochastic Optimization

It is well known that we need to choose the hyper-parameters in Momentum, AdaGrad, AdaDelta, and other alternative stochastic optimizers. While in many cases, the hyper-parameters are tuned tediously based on experience becoming more of an art than science. We present a novel per-dimension learning rate method for gradient descent called AdaSmooth. The method is insensitive to hyper-parameters thus it requires no manual tuning of the hyper-parameters like Momentum, AdaGrad, and AdaDelta methods. We show promising results compared to other methods on different convolutional neural networks, multi-layer perceptron, and alternative machine learning tasks. Empirical results demonstrate that AdaSmooth works well in practice and compares favorably to other stochastic optimization methods in neural networks.
Intelligence at the Extreme Edge: A Survey on Reformable TinyML
Authors: Visal Rajapakse, Ishan Karunanayake, Nadeem Ahmed
Date: 2022-04-02 00:00:00
Tasks: Benchmarking, BIG-bench Machine Learning, Survey

Tiny Machine Learning (TinyML) is an upsurging research field that proposes to democratize the use of Machine Learning and Deep Learning on highly energy-efficient frugal Microcontroller Units. Considering the general assumption that TinyML can only run inference, growing interest in the domain has led to work that makes them reformable, i.e., solutions that permit models to improve once deployed. This work presents a survey on reformable TinyML solutions with the proposal of a novel taxonomy. Here, the suitability of each hierarchical layer for reformability is discussed. Furthermore, we explore the workflow of TinyML and analyze the identified deployment schemes, available tools and the scarcely available benchmarking tools. Finally, we discuss how reformable TinyML can impact a few selected industrial areas and discuss the challenges and future directions.
Chordal Sparsity for Lipschitz Constant Estimation of Deep Neural Networks
Authors: Anton Xue, Lars Lindemann, Alexander Robey, Hamed Hassani, George J. Pappas, Rajeev Alur
Date: 2022-04-02 00:00:00
Tasks: image-classification, Image Classification, Navigate

Lipschitz constants of neural networks allow for guarantees of robustness in image classification, safety in controller design, and generalizability beyond the training data. As calculating Lipschitz constants is NP-hard, techniques for estimating Lipschitz constants must navigate the trade-off between scalability and accuracy. In this work, we significantly push the scalability frontier of a semidefinite programming technique known as LipSDP while achieving zero accuracy loss. We first show that LipSDP has chordal sparsity, which allows us to derive a chordally sparse formulation that we call Chordal-LipSDP. The key benefit is that the main computational bottleneck of LipSDP, a large semidefinite constraint, is now decomposed into an equivalent collection of smaller ones: allowing Chordal-LipSDP to outperform LipSDP particularly as the network depth grows. Moreover, our formulation uses a tunable sparsity parameter that enables one to gain tighter estimates without incurring a significant computational cost. We illustrate the scalability of our approach through extensive numerical experiments.
A Differential Evolution-Enhanced Latent Factor Analysis Model for High-dimensional and Sparse Data
Authors: Jia Chen, Di wu, Xin Luo
Date: 2022-04-02 00:00:00
Tasks: Position

High-dimensional and sparse (HiDS) matrices are frequently adopted to describe the complex relationships in various big data-related systems and applications. A Position-transitional Latent Factor Analysis (PLFA) model can accurately and efficiently represent an HiDS matrix. However, its involved latent factors are optimized by stochastic gradient descent with the specific gradient direction step-by-step, which may cause a suboptimal solution. To address this issue, this paper proposes a Sequential-Group-Differential- Evolution (SGDE) algorithm to refine the latent factors optimized by a PLFA model, thereby achieving a highly-accurate SGDE-PLFA model to HiDS matrices. As demonstrated by the experiments on four HiDS matrices, a SGDE-PLFA model outperforms the state-of-the-art models.
Learning List-wise Representation in Reinforcement Learning for Ads Allocation with Multiple Auxiliary Tasks
Authors: Ze Wang, Guogang Liao, Xiaowen Shi, Xiaoxu Wu, Chuheng Zhang, Yongkang Wang, Xingxing Wang, Dong Wang
Date: 2022-04-02 00:00:00
Tasks: Contrastive Learning, Reinforcement Learning (RL)

With the recent prevalence of reinforcement learning (RL), there have been tremendous interests in utilizing RL for ads allocation in recommendation platforms (e.g., e-commerce and news feed sites). To achieve better allocation, the input of recent RL-based ads allocation methods is upgraded from point-wise single item to list-wise item arrangement. However, this also results in a high-dimensional space of state-action pairs, making it difficult to learn list-wise representations with good generalization ability. This further hinders the exploration of RL agents and causes poor sample efficiency. To address this problem, we propose a novel RL-based approach for ads allocation which learns better list-wise representations by leveraging task-specific signals on Meituan food delivery platform. Specifically, we propose three different auxiliary tasks based on reconstruction, prediction, and contrastive learning respectively according to prior domain knowledge on ads allocation. We conduct extensive experiments on Meituan food delivery platform to evaluate the effectiveness of the proposed auxiliary tasks. Both offline and online experimental results show that the proposed method can learn better list-wise representations and achieve higher revenue for the platform compared to the state-of-the-art baselines.
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation
Authors: Minsoo Kang, Jaeyoo Park, Bohyung Han
Date: 2022-04-02 00:00:00
Proceeding: CVPR 2022 1
Tasks: class-incremental learning, Class Incremental Learning, Incremental Learning, Knowledge Distillation

We present a novel class incremental learning approach based on deep neural networks, which continually learns new tasks with limited memory for storing examples in the previous tasks. Our algorithm is based on knowledge distillation and provides a principled way to maintain the representations of old models while adjusting to new tasks effectively. The proposed method estimates the relationship between the representation changes and the resulting loss increases incurred by model updates. It minimizes the upper bound of the loss increases using the representations, which exploits the estimated importance of each feature map within a backbone model. Based on the importance, the model restricts updates of important features for robustness while allowing changes in less critical features for flexibility. This optimization strategy effectively alleviates the notorious catastrophic forgetting problem despite the limited accessibility of data in the previous tasks. The experimental results show significant accuracy improvement of the proposed algorithm over the existing methods on the standard datasets. Code is available.
AutoProtoNet: Interpretability for Prototypical Networks
Authors: Pedro Sandoval-Segura, Wallace Lawson
Date: 2022-04-02 00:00:00
Tasks: Few-Shot Learning, Meta-Learning

In meta-learning approaches, it is difficult for a practitioner to make sense of what kind of representations the model employs. Without this ability, it can be difficult to both understand what the model knows as well as to make meaningful corrections. To address these challenges, we introduce AutoProtoNet, which builds interpretability into Prototypical Networks by training an embedding space suitable for reconstructing inputs, while remaining convenient for few-shot learning. We demonstrate how points in this embedding space can be visualized and used to understand class representations. We also devise a prototype refinement method, which allows a human to debug inadequate classification parameters. We use this debugging technique on a custom classification task and find that it leads to accuracy improvements on a validation set consisting of in-the-wild images. We advocate for interpretability in meta-learning approaches and show that there are interactive ways for a human to enhance meta-learning algorithms.
Long-tailed Extreme Multi-label Text Classification with Generated Pseudo Label Descriptions
Authors: Ruohong Zhang, Yau-Shian Wang, Yiming Yang, Donghan Yu, Tom Vu, Likun Lei
Date: 2022-04-02 00:00:00
Tasks: Multi Label Text Classification, Multi-Label Text Classification, Pseudo Label, Retrieval, text-classification, Text Classification

Extreme Multi-label Text Classification (XMTC) has been a tough challenge in machine learning research and applications due to the sheer sizes of the label spaces and the severe data scarce problem associated with the long tail of rare labels in highly skewed distributions. This paper addresses the challenge of tail label prediction by proposing a novel approach, which combines the effectiveness of a trained bag-of-words (BoW) classifier in generating informative label descriptions under severe data scarce conditions, and the power of neural embedding based retrieval models in mapping input documents (as queries) to relevant label descriptions. The proposed approach achieves state-of-the-art performance on XMTC benchmark datasets and significantly outperforms the best methods so far in the tail label prediction. We also provide a theoretical analysis for relating the BoW and neural models w.r.t. performance lower bound.
Enhancing Digital Health Services: A Machine Learning Approach to Personalized Exercise Goal Setting
Authors: Ji Fang, Vincent CS Lee, Hao Ji, HaiYan Wang
Date: 2022-04-03 00:00:00
Tasks: Deep Reinforcement Learning, Management, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Time Series Analysis

The utilization of digital health has increased recently, and these services provide extensive guidance to encourage users to exercise frequently by setting daily exercise goals to promote a healthy lifestyle. These comprehensive guides evolved from the consideration of various personalized behavioral factors. Nevertheless, existing approaches frequently neglect the users dynamic behavior and the changing in their health conditions. This study aims to fill this gap by developing a machine learning algorithm that dynamically updates auto-suggestion exercise goals using retrospective data and realistic behavior trajectory. We conducted a methodological study by designing a deep reinforcement learning algorithm to evaluate exercise performance, considering fitness-fatigue effects. The deep reinforcement learning algorithm combines deep learning techniques to analyse time series data and infer user exercise behavior. In addition, we use the asynchronous advantage actor-critic algorithm for reinforcement learning to determine the optimal exercise intensity through exploration and exploitation. The personalized exercise data and biometric data used in this study were collected from publicly available datasets, encompassing walking, sports logs, and running. In our study, we conducted The statistical analyses/inferential tests to compare the effectiveness of machine learning approach in exercise goal setting across different exercise goal setting strategies.
FedGBF: An efficient vertical federated learning framework via gradient boosting and bagging
Authors: Yujin Han, Pan Du, Kai Yang
Date: 2022-04-03 00:00:00
Tasks: Federated Learning, Vertical Federated Learning

Federated learning, conducive to solving data privacy and security problems, has attracted increasing attention recently. However, the existing federated boosting model sequentially builds a decision tree model with the weak base learner, resulting in redundant boosting steps and high interactive communication costs. In contrast, the federated bagging model saves time by building multi-decision trees in parallel, but it suffers from performance loss. With the aim of obtaining an outstanding performance with less time cost, we propose a novel model in a vertically federated setting termed as Federated Gradient Boosting Forest (FedGBF). FedGBF simultaneously integrates the boosting and bagging's preponderance by building the decision trees in parallel as a base learner for boosting. Subsequent to FedGBF, the problem of hyperparameters tuning is rising. Then we propose the Dynamic FedGBF, which dynamically changes each forest's parameters and thus reduces the complexity. Finally, the experiments based on the benchmark datasets demonstrate the superiority of our method.
Breaking the De-Pois Poisoning Defense
Authors: Alaa Anani, Mohamed Ghanem, Lotfy Abdel Khaliq
Date: 2022-04-03 00:00:00

Attacks on machine learning models have been, since their conception, a very persistent and evasive issue resembling an endless cat-and-mouse game. One major variant of such attacks is poisoning attacks which can indirectly manipulate an ML model. It has been observed over the years that the majority of proposed effective defense models are only effective when an attacker is not aware of them being employed. In this paper, we show that the attack-agnostic De-Pois defense is hardly an exception to that rule. In fact, we demonstrate its vulnerability to the simplest White-Box and Black-Box attacks by an attacker that knows the structure of the De-Pois defense model. In essence, the De-Pois defense relies on a critic model that can be used to detect poisoned data before passing it to the target model. In our work, we break this poison-protection layer by replicating the critic model and then performing a composed gradient-sign attack on both the critic and target models simultaneously -- allowing us to bypass the critic firewall to poison the target model.
pmuBAGE: The Benchmarking Assortment of Generated PMU Data for Power System Events -- Part I: Overview and Results
Authors: Brandon Foggo, Koji Yamashita, Nanpeng Yu
Date: 2022-04-03 00:00:00
Tasks: Benchmarking

We present pmuGE (phasor measurement unit Generator of Events), one of the first data-driven generative model for power system event data. We have trained this model on thousands of actual events and created a dataset denoted pmuBAGE (the Benchmarking Assortment of Generated PMU Events). The dataset consists of almost 1000 instances of labeled event data to encourage benchmark evaluations on phasor measurement unit (PMU) data analytics. The dataset is available online for use by any researcher or practitioner in the field. PMU data are challenging to obtain, especially those covering event periods. Nevertheless, power system problems have recently seen phenomenal advancements via data-driven machine learning solutions - solutions created by researchers who were fortunate enough to obtain such PMU data. A highly accessible standard benchmarking dataset would enable a drastic acceleration of the development of successful machine learning techniques in this field. We propose a novel learning method based on the Event Participation Decomposition of Power System Events, which makes it possible to learn a generative model of PMU data during system anomalies. The model can create highly realistic event data without compromising the differential privacy of the PMUs used to train it. The dataset is available online for any researcher to use at the pmuBAGE Github Repository - https://github.com/NanpengYu/pmuBAGE. Part I - This is part I of a two part paper. In part I, we describe a high level overview of pmuBAGE, its creation, and the experiments used to test it. Part II will discuss the exact models used in its generation in far more detail.
Seemo: A new tool for early design window view satisfaction evaluation in residential buildings
Authors: Jaeha Kim, Michael Kent, Katharina Kral, Timur Dogan
Date: 2022-04-03 00:00:00
Tasks: Prediction

People spend approximately 90% of their lives indoors, and thus arguably, the indoor space design can significantly influence occupant well-being. Adequate views to the outside are one of the most cited indoor qualities related to occupant well-being. However, due to urbanization and densification trends, designers may have difficulties in providing vistas and views to the outside with an assortment of content, which can support the needs of their occupants. To better understand occupant view satisfaction and provide reliable design feedback to architects, existing view satisfaction data must be expanded to capture a wider variety of view scenarios and occupants. Most related research remains challenging in architectural practice due to a lack of easy-to-use early-design analysis tools. However, early assessment of view can be advantageous as design decisions in early design, such as building orientation, plan layout, and facade design, can improve the view quality. This paper, hence, presents results from a 181 participant view satisfaction survey with 590 window views. The survey data is used to train a tree-regression model to predict view satisfaction. The prediction performance was compared to an existing view assessment framework through case studies. The result showed that the new prediction is more accurate to the surveyed result than the framework. Further, the prediction performance was generally high for most responses, verifying the reliability. To facilitate view analysis in early design, this paper describes integrating the satisfaction prediction model and a ray-casting tool to compute view parameters in the CAD environment.
Model-Parallel Fourier Neural Operators as Learned Surrogates for Large-Scale Parametric PDEs
Authors: Thomas J. Grady II, Rishi Khan, Mathias Louboutin, Ziyi Yin, Philipp A. Witte, Ranveer Chandra, Russell J. Hewett, Felix J. Herrmann
Date: 2022-04-04 00:00:00

Fourier neural operators (FNOs) are a recently introduced neural network architecture for learning solution operators of partial differential equations (PDEs), which have been shown to perform significantly better than comparable deep learning approaches. Once trained, FNOs can achieve speed-ups of multiple orders of magnitude over conventional numerical PDE solvers. However, due to the high dimensionality of their input data and network weights, FNOs have so far only been applied to two-dimensional or small three-dimensional problems. To remove this limited problem-size barrier, we propose a model-parallel version of FNOs based on domain-decomposition of both the input data and network weights. We demonstrate that our model-parallel FNO is able to predict time-varying PDE solutions of over 2.6 billion variables on Perlmutter using up to 512 A100 GPUs and show an example of training a distributed FNO on the Azure cloud for simulating multiphase CO$_2$ dynamics in the Earth's subsurface.
FedSynth: Gradient Compression via Synthetic Data in Federated Learning
Authors: Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, Yue Liu
Date: 2022-04-04 00:00:00
Tasks: Federated Learning, Model Compression

Model compression is important in federated learning (FL) with large models to reduce communication cost. Prior works have been focusing on sparsification based compression that could desparately affect the global model accuracy. In this work, we propose a new scheme for upstream communication where instead of transmitting the model update, each client learns and transmits a light-weight synthetic dataset such that using it as the training data, the model performs similarly well on the real training data. The server will recover the local model update via the synthetic data and apply standard aggregation. We then provide a new algorithm FedSynth to learn the synthetic data locally. Empirically, we find our method is comparable/better than random masking baselines in all three common federated learning benchmark datasets.
Empirical Analysis of Lifelog Data using Optimal Feature Selection based Unsupervised Logistic Regression (OFS-ULR) Model with Spark Streaming
Authors: Sadhana Tiwari, Sonali Agarwal
Date: 2022-04-04 00:00:00
Tasks: feature selection, Time Series Analysis

Recent advancement in the field of pervasive healthcare monitoring systems causes the generation of a huge amount of lifelog data in real-time. Chronic diseases are one of the most serious health challenges in developing and developed countries. According to WHO, this accounts for 73% of all deaths and 60% of the global burden of diseases. Chronic disease classification models are now harnessing the potential of lifelog data to explore better healthcare practices. This paper is to construct an optimal feature selection-based unsupervised logistic regression model (OFS-ULR) to classify chronic diseases. Since lifelog data analysis is crucial due to its sensitive nature; thus the conventional classification models show limited performance. Therefore, designing new classifiers for the classification of chronic diseases using lifelog data is the need of the age. The vital part of building a good model depends on pre-processing of the dataset, identifying important features, and then training a learning algorithm with suitable hyper parameters for better performance. The proposed approach improves the performance of existing methods using a series of steps such as (i) removing redundant or invalid instances, (ii) making the data labelled using clustering and partitioning the data into classes, (iii) identifying the suitable subset of features by applying either some domain knowledge or selection algorithm, (iv) hyper parameter tuning for models to get best results, and (v) performance evaluation using Spark streaming environment. For this purpose, two-time series datasets are used in the experiment to compute the accuracy, recall, precision, and f1-score. The experimental analysis proves the suitability of the proposed approach as compared to the conventional classifiers and our newly constructed model achieved highest accuracy and reduced training complexity among all among all.