pwc-archive/papers-with-abstracts huggingface dataset:
NGAME: Negative Mining-aware Mini-batching for Extreme Classification
Authors: Kunal Dahiya, Nilesh Gupta, Deepak Saini, Akshay Soni, Yajun Wang, Kushal Dave, Jian Jiao, Gururaj K, Prasenjit Dey, Amit Singh, Deepesh Hada, Vidit Jain, Bhawna Paliwal, Anshul Mittal, Sonu Mehta, Ramachandran Ramjee, Sumeet Agarwal, Purushottam Kar, Manik Varma
Date: 2022-07-10 00:00:00
Tasks: Classification, TAG

Extreme Classification (XC) seeks to tag data points with the most relevant subset of labels from an extremely large label set. Performing deep XC with dense, learnt representations for data points and labels has attracted much attention due to its superiority over earlier XC methods that used sparse, hand-crafted features. Negative mining techniques have emerged as a critical component of all deep XC methods that allow them to scale to millions of labels. However, despite recent advances, training deep XC models with large encoder architectures such as transformers remains challenging. This paper identifies that memory overheads of popular negative mining techniques often force mini-batch sizes to remain small and slow training down. In response, this paper introduces NGAME, a light-weight mini-batch creation technique that offers provably accurate in-batch negative samples. This allows training with larger mini-batches offering significantly faster convergence and higher accuracies than existing negative sampling techniques. NGAME was found to be up to 16% more accurate than state-of-the-art methods on a wide array of benchmark datasets for extreme classification, as well as 3% more accurate at retrieving search engine queries in response to a user webpage visit to show personalized ads. In live A/B tests on a popular search engine, NGAME yielded up to 23% gains in click-through-rates.
On a hypercycle equation with infinitely many members
Authors: Alexander S. Bratus, Olga S. Chmereva, Ivan Yegorov, Artem S. Novozhilov
Date: 2022-07-11 00:00:00

A hypercycle equation with infinitely many types of macromolecules is formulated and studied both analytically and numerically. The resulting model is given by an integro-differential equation of the mixed type. Sufficient conditions for the existence, uniqueness, and non-negativity of solutions are formulated and proved. Analytical evidence is provided for the existence of non-uniform (with respect to the second variable) steady states. Finally, numerical simulations strongly indicate the existence of a stable nonlinear wave in the form of the wave train.
An improved Epidemiological-Unscented Kalman Filter (Hybrid SEIHCRDV-UKF) model for the prediction of COVID-19. Application on real-time data
Authors: Vasileios E. Papageorgiou, George Tsaklidis
Date: 2022-07-03 00:00:00

The prevalence of COVID-19 has been the most serious health challenge of the 21th century to date, concerning national health systems on a daily basis, since December 2019 when it appeared in Wuhan City. Nevertheless, most of the proposed mathematical methodologies aiming to describe the dynamics of an epidemic, rely on deterministic models that are not able to reflect the true nature of its spread. In this paper, we propose a SEIHCRDV model - an extension/improvement of the classic SIR compartmental model - which also takes into consideration the populations of exposed, hospitalized, admitted in intensive care units (ICU), deceased and vaccinated cases, in combination with an unscented Kalman filter (UKF), providing a dynamic estimation of the time dependent system's parameters. The stochastic approach is considered necessary, as both observations and system equations are characterized by uncertainties. Apparently, this new consideration is useful for examining various pandemics more effectively. The reliability of the model is examined on the daily recordings of COVID-19 in France, over a long period of 265 days. Two major waves of infection are observed, starting in January 2021, which signified the start of vaccinations in Europe providing quite encouraging predictive performance, based on the produced NRMSE values. Special emphasis is placed on proving the non-negativity of SEIHCRDV model, achieving a representative basic reproductive number R0 and demonstrating the existence and stability of disease equilibria according to the formula produced to estimate R0. The model outperforms in predictive ability not only deterministic approaches but also state-of-the-art stochastic models that employ Kalman filters.
Bio-inspired Machine Learning: programmed death and replication
Authors: Andrey Grabovsky, Vitaly Vanchurin
Date: 2022-06-30 00:00:00
Tasks: BIG-bench Machine Learning

We analyze algorithmic and computational aspects of biological phenomena, such as replication and programmed death, in the context of machine learning. We use two different measures of neuron efficiency to develop machine learning algorithms for adding neurons to the system (i.e. replication algorithm) and removing neurons from the system (i.e. programmed death algorithm). We argue that the programmed death algorithm can be used for compression of neural networks and the replication algorithm can be used for improving performance of the already trained neural networks. We also show that a combined algorithm of programmed death and replication can improve the learning efficiency of arbitrary machine learning systems. The computational advantages of the bio-inspired algorithms are demonstrated by training feedforward neural networks on the MNIST dataset of handwritten images.
Bregman Proximal Langevin Monte Carlo via Bregman--Moreau Envelopes
Authors: Tim Tsz-Kit Lau, Han Liu
Date: 2022-07-10 00:00:00

We propose efficient Langevin Monte Carlo algorithms for sampling distributions with nonsmooth convex composite potentials, which is the sum of a continuously differentiable function and a possibly nonsmooth function. We devise such algorithms leveraging recent advances in convex analysis and optimization methods involving Bregman divergences, namely the Bregman--Moreau envelopes and the Bregman proximity operators, and in the Langevin Monte Carlo algorithms reminiscent of mirror descent. The proposed algorithms extend existing Langevin Monte Carlo algorithms in two aspects -- the ability to sample nonsmooth distributions with mirror descent-like algorithms, and the use of the more general Bregman--Moreau envelope in place of the Moreau envelope as a smooth approximation of the nonsmooth part of the potential. A particular case of the proposed scheme is reminiscent of the Bregman proximal gradient algorithm. The efficiency of the proposed methodology is illustrated with various sampling tasks at which existing Langevin Monte Carlo methods are known to perform poorly.
Domain Adaptation Under Behavioral and Temporal Shifts for Natural Time Series Mobile Activity Recognition
Authors: Garrett Wilson, Janardhan Rao Doppa, Diane J. Cook
Date: 2022-07-10 00:00:00
Tasks: Activity Recognition, Contrastive Learning, Domain Adaptation, Human Activity Recognition, Time Series, Time Series Analysis, Unsupervised Domain Adaptation

Increasingly, human behavior is captured on mobile devices, leading to an increased interest in automated human activity recognition. However, existing datasets typically consist of scripted movements. Our long-term goal is to perform mobile activity recognition in natural settings. We collect a dataset to support this goal with activity categories that are relevant for downstream tasks such as health monitoring and intervention. Because of the large variations present in human behavior, we collect data from many participants across two different age groups. Because human behavior can change over time, we also collect data from participants over a month's time to capture the temporal drift. We hypothesize that mobile activity recognition can benefit from unsupervised domain adaptation algorithms. To address this need and test this hypothesis, we analyze the performance of domain adaptation across people and across time. We then enhance unsupervised domain adaptation with contrastive learning and with weak supervision when label proportions are available. The dataset is available at https://github.com/WSU-CASAS/smartwatch-data
Emerging Patterns in the Continuum Representation of Protein-Lipid Fingerprints
Authors: Konstantia Georgouli, Helgi I Ingólfsson, Fikret Aydin, Mark Heimann, Felice C Lightstone, Peer-Timo Bremer, Harsh Bhatia
Date: 2022-07-09 00:00:00
Tasks: Descriptive

Capturing intricate biological phenomena often requires multiscale modeling where coarse and inexpensive models are developed using limited components of expensive and high-fidelity models. Here, we consider such a multiscale framework in the context of cancer biology and address the challenge of evaluating the descriptive capabilities of a continuum model developed using 1-dimensional statistics from a molecular dynamics model. Using deep learning, we develop a highly predictive classification model that identifies complex and emergent behavior from the continuum model. With over 99.9% accuracy demonstrated for two simulations, our approach confirms the existence of protein-specific "lipid fingerprints", i.e. spatial rearrangements of lipids in response to proteins of interest. Through this demonstration, our model also provides external validation of the continuum model, affirms the value of such multiscale modeling, and can foster new insights through further analysis of these fingerprints.
Dynamic Time Warping based Adversarial Framework for Time-Series Domain
Authors: Taha Belkhouja, Yan Yan, Janardhan Rao Doppa
Date: 2022-07-09 00:00:00
Tasks: Adversarial Robustness, Dynamic Time Warping, Time Series, Time Series Analysis

Despite the rapid progress on research in adversarial robustness of deep neural networks (DNNs), there is little principled work for the time-series domain. Since time-series data arises in diverse applications including mobile health, finance, and smart grid, it is important to verify and improve the robustness of DNNs for the time-series domain. In this paper, we propose a novel framework for the time-series domain referred as {\em Dynamic Time Warping for Adversarial Robustness (DTW-AR)} using the dynamic time warping measure. Theoretical and empirical evidence is provided to demonstrate the effectiveness of DTW over the standard Euclidean distance metric employed in prior methods for the image domain. We develop a principled algorithm justified by theoretical analysis to efficiently create diverse adversarial examples using random alignment paths. Experiments on diverse real-world benchmarks show the effectiveness of DTW-AR to fool DNNs for time-series data and to improve their robustness using adversarial training. The source code of DTW-AR algorithms is available at https://github.com/tahabelkhouja/DTW-AR
Adversarial Framework with Certified Robustness for Time-Series Domain via Statistical Features
Authors: Taha Belkhouja, Janardhan Rao Doppa
Date: 2022-07-09 00:00:00
Tasks: Time Series, Time Series Analysis

Time-series data arises in many real-world applications (e.g., mobile health) and deep neural networks (DNNs) have shown great success in solving them. Despite their success, little is known about their robustness to adversarial attacks. In this paper, we propose a novel adversarial framework referred to as Time-Series Attacks via STATistical Features (TSA-STAT)}. To address the unique challenges of time-series domain, TSA-STAT employs constraints on statistical features of the time-series data to construct adversarial examples. Optimized polynomial transformations are used to create attacks that are more effective (in terms of successfully fooling DNNs) than those based on additive perturbations. We also provide certified bounds on the norm of the statistical features for constructing adversarial examples. Our experiments on diverse real-world benchmark datasets show the effectiveness of TSA-STAT in fooling DNNs for time-series domain and in improving their robustness. The source code of TSA-STAT algorithms is available at https://github.com/tahabelkhouja/Time-Series-Attacks-via-STATistical-Features
Out-of-Distribution Detection in Time-Series Domain: A Novel Seasonal Ratio Scoring Approach
Authors: Taha Belkhouja, Yan Yan, Janardhan Rao Doppa
Date: 2022-07-09 00:00:00
Tasks: Out-of-Distribution Detection, Out of Distribution (OOD) Detection, Time Series, Time Series Analysis

Safe deployment of time-series classifiers for real-world applications relies on the ability to detect the data which is not generated from the same distribution as training data. This task is referred to as out-of-distribution (OOD) detection. We consider the novel problem of OOD detection for the time-series domain. We discuss the unique challenges posed by time-series data and explain why prior methods from the image domain will perform poorly. Motivated by these challenges, this paper proposes a novel {\em Seasonal Ratio Scoring (SRS)} approach. SRS consists of three key algorithmic steps. First, each input is decomposed into class-wise semantic component and remainder. Second, this decomposition is employed to estimate the class-wise conditional likelihoods of the input and remainder using deep generative models. The seasonal ratio score is computed from these estimates. Third, a threshold interval is identified from the in-distribution data to detect OOD examples. Experiments on diverse real-world benchmarks demonstrate that the SRS method is well-suited for time-series OOD detection when compared to baseline methods. Open-source code for SRS method is provided at https://github.com/tahabelkhouja/SRS
Graph-based Molecular Representation Learning
Authors: Zhichun Guo, Kehan Guo, Bozhao Nan, Yijun Tian, Roshni G. Iyer, Yihong Ma, Olaf Wiest, Xiangliang Zhang, Wei Wang, Chuxu Zhang, Nitesh V. Chawla
Date: 2022-07-08 00:00:00
Tasks: Graph Learning, molecular representation, Molecule Captioning, Property Prediction, Representation Learning

Molecular representation learning (MRL) is a key step to build the connection between machine learning and chemical science. In particular, it encodes molecules as numerical vectors preserving the molecular structures and features, on top of which the downstream tasks (e.g., property prediction) can be performed. Recently, MRL has achieved considerable progress, especially in methods based on deep molecular graph learning. In this survey, we systematically review these graph-based molecular representation techniques, especially the methods incorporating chemical domain knowledge. Specifically, we first introduce the features of 2D and 3D molecular graphs. Then we summarize and categorize MRL methods into three groups based on their input. Furthermore, we discuss some typical chemical applications supported by MRL. To facilitate studies in this fast-developing area, we also list the benchmarks and commonly used datasets in the paper. Finally, we share our thoughts on future research directions.
Training Robust Deep Models for Time-Series Domain: Novel Algorithms and Theoretical Analysis
Authors: Taha Belkhouja, Yan Yan, Janardhan Rao Doppa
Date: 2022-07-09 00:00:00
Tasks: Data Augmentation, Dynamic Time Warping, Time Series, Time Series Analysis, Time Series Classification

Despite the success of deep neural networks (DNNs) for real-world applications over time-series data such as mobile health, little is known about how to train robust DNNs for time-series domain due to its unique characteristics compared to images and text data. In this paper, we propose a novel algorithmic framework referred as RObust Training for Time-Series (RO-TS) to create robust DNNs for time-series classification tasks. Specifically, we formulate a min-max optimization problem over the model parameters by explicitly reasoning about the robustness criteria in terms of additive perturbations to time-series inputs measured by the global alignment kernel (GAK) based distance. We also show the generality and advantages of our formulation using the summation structure over time-series alignments by relating both GAK and dynamic time warping (DTW). This problem is an instance of a family of compositional min-max optimization problems, which are challenging and open with unclear theoretical guarantee. We propose a principled stochastic compositional alternating gradient descent ascent (SCAGDA) algorithm for this family of optimization problems. Unlike traditional methods for time-series that require approximate computation of distance measures, SCAGDA approximates the GAK based distance on-the-fly using a moving average approach. We theoretically analyze the convergence rate of SCAGDA and provide strong theoretical support for the estimation of GAK based distance. Our experiments on real-world benchmarks demonstrate that RO-TS creates more robust DNNs when compared to adversarial training using prior methods that rely on data augmentation or new definitions of loss functions. We also demonstrate the importance of GAK for time-series data over the Euclidean distance. The source code of RO-TS algorithms is available at https://github.com/tahabelkhouja/Robust-Training-for-Time-Series
Attention and Self-Attention in Random Forests
Authors: Lev V. Utkin, Andrei V. Konstantinov
Date: 2022-07-09 00:00:00
Tasks: regression

New models of random forests jointly using the attention and self-attention mechanisms are proposed for solving the regression problem. The models can be regarded as extensions of the attention-based random forest whose idea stems from applying a combination of the Nadaraya-Watson kernel regression and the Huber's contamination model to random forests. The self-attention aims to capture dependencies of the tree predictions and to remove noise or anomalous predictions in the random forest. The self-attention module is trained jointly with the attention module for computing weights. It is shown that the training process of attention weights is reduced to solving a single quadratic or linear optimization problem. Three modifications of the general approach are proposed and compared. A specific multi-head self-attention for the random forest is also considered. Heads of the self-attention are obtained by changing its tuning parameters including the kernel parameters and the contamination parameter of models. Numerical experiments with various datasets illustrate the proposed models and show that the supplement of the self-attention improves the model performance for many datasets.
Fuzzy Clustering by Hyperbolic Smoothing
Authors: David Masis, Esteban Segura, Javier Trejos, Adilson Xavier
Date: 2022-07-09 00:00:00
Tasks: Clustering

We propose a novel method for building fuzzy clusters of large data sets, using a smoothing numerical approach. The usual sum-of-squares criterion is relaxed so the search for good fuzzy partitions is made on a continuous space, rather than a combinatorial space as in classical methods \cite{Hartigan}. The smoothing allows a conversion from a strongly non-differentiable problem into differentiable subproblems of optimization without constraints of low dimension, by using a differentiable function of infinite class. For the implementation of the algorithm we used the statistical software $R$ and the results obtained were compared to the traditional fuzzy $C$--means method, proposed by Bezdek.
A Statistical-Modelling Approach to Feedforward Neural Network Model Selection
Authors: Andrew McInerney, Kevin Burke
Date: 2022-07-09 00:00:00
Tasks: Model Selection, Variable Selection

Feedforward neural networks (FNNs) can be viewed as non-linear regression models, where covariates enter the model through a combination of weighted summations and non-linear functions. Although these models have some similarities to the approaches used within statistical modelling, the majority of neural network research has been conducted outside of the field of statistics. This has resulted in a lack of statistically-based methodology, and, in particular, there has been little emphasis on model parsimony. Determining the input layer structure is analogous to variable selection, while the structure for the hidden layer relates to model complexity. In practice, neural network model selection is often carried out by comparing models using out-of-sample performance. However, in contrast, the construction of an associated likelihood function opens the door to information-criteria-based variable and architecture selection. A novel model selection method, which performs both input- and hidden-node selection, is proposed using the Bayesian information criterion (BIC) for FNNs. The choice of BIC over out-of-sample performance as the model selection objective function leads to an increased probability of recovering the true model, while parsimoniously achieving favourable out-of-sample performance. Simulation studies are used to evaluate and justify the proposed method, and applications on real data are investigated.
On the Robustness and Anomaly Detection of Sparse Neural Networks
Authors: Morgane Ayle, Bertrand Charpentier, John Rachwan, Daniel Zügner, Simon Geisler, Stephan Günnemann
Date: 2022-07-09 00:00:00
Tasks: Anomaly Detection

The robustness and anomaly detection capability of neural networks are crucial topics for their safe adoption in the real-world. Moreover, the over-parameterization of recent networks comes with high computational costs and raises questions about its influence on robustness and anomaly detection. In this work, we show that sparsity can make networks more robust and better anomaly detectors. To motivate this even further, we show that a pre-trained neural network contains, within its parameter space, sparse subnetworks that are better at these tasks without any further training. We also show that structured sparsity greatly helps in reducing the complexity of expensive robustness and detection methods, while maintaining or even improving their results on these tasks. Finally, we introduce a new method, SensNorm, which uses the sensitivity of weights derived from an appropriate pruning method to detect anomalous samples in the input.
Multi-label Classification with High-rank and High-order Label Correlations
Authors: Chongjie Si, Yuheng Jia, Ran Wang, Min-Ling Zhang, Yanghe Feng, Chongxiao Qu
Date: 2022-07-09 00:00:00
Tasks: Common Sense Reasoning, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, Vocal Bursts Intensity Prediction

Exploiting label correlations is important to multi-label classification. Previous methods capture the high-order label correlations mainly by transforming the label matrix to a latent label space with low-rank matrix factorization. However, the label matrix is generally a full-rank or approximate full-rank matrix, making the low-rank factorization inappropriate. Besides, in the latent space, the label correlations will become implicit. To this end, we propose a simple yet effective method to depict the high-order label correlations explicitly, and at the same time maintain the high-rank of the label matrix. Moreover, we estimate the label correlations and infer model parameters simultaneously via the local geometric structure of the input to achieve mutual enhancement. Comparative studies over twelve benchmark data sets validate the effectiveness of the proposed algorithm in multi-label classification. The exploited high-order label correlations are consistent with common sense empirically. Our code is publicly available at https://github.com/Chongjie-Si/HOMI.