pwc-archive/papers-with-abstracts huggingface dataset:
The Mean Dimension of Neural Networks -- What causes the interaction effects?
Authors: Roman Hahn, Christoph Feinauer, Emanuele Borgonovo
Date: 2022-07-11 00:00:00

Owen and Hoyt recently showed that the effective dimension offers key structural information about the input-output mapping underlying an artificial neural network. Along this line of research, this work proposes an estimation procedure that allows the calculation of the mean dimension from a given dataset, without resampling from external distributions. The design yields total indices when features are independent and a variant of total indices when features are correlated. We show that this variant possesses the zero independence property. With synthetic datasets, we analyse how the mean dimension evolves layer by layer and how the activation function impacts the magnitude of interactions. We then use the mean dimension to study some of the most widely employed convolutional architectures for image recognition (LeNet, ResNet, DenseNet). To account for pixel correlations, we propose calculating the mean dimension after the addition of an inverse PCA layer that allows one to work on uncorrelated PCA-transformed features, without the need to retrain the neural network. We use the generalized total indices to produce heatmaps for post-hoc explanations, and we employ the mean dimension on the PCA-transformed features for cross comparisons of the artificial neural networks structures. Results provide several insights on the difference in magnitude of interactions across the architectures, as well as indications on how the mean dimension evolves during training.
RRMSE Voting Regressor: A weighting function based improvement to ensemble regression
Authors: Shikun Chen, Nguyen Manh Luc
Date: 2022-07-11 00:00:00
Tasks: regression

This paper describes the RRMSE (Relative Root Mean Square Error) based weights to weight the occurrences of predictive values before averaging for the ensemble voting regression. The core idea behind ensemble regression is to combine several base regression models in order to improve the prediction performance in learning problems with a numeric continuous target variable. The default weights setting for the ensemble voting regression is uniform weights, and without domain knowledge of learning task, assigning weights for predictions are impossible, which makes it very difficult to improve the predictions. This work attempts to improve the prediction of voting regression by implementing the RRMSE based weighting function. Experiments show that RRMSE voting regressor produces significantly better predictions than other state-of-the-art ensemble regression algorithms on six popular regression learning datasets.
BLANT: Basic Local Alignment of Network Topology, Part 1: Seeding local alignments with unambiguous 8-node graphlets
Authors: Patrick Wang, Henry Ye, Wayne B Hayes
Date: 2022-07-10 00:00:00
Tasks: Specificity

BLAST is a standard tool in bioinformatics for creating local sequence alignments using a "seed-and-extend" approach. Here we introduce an analogous seed-and-extend algorithm that produces local network alignments: BLANT, for Basic Local Alignment of Network Topology. This paper introduces BLANT-seed: given an input graph, BLANT-seed uses network topology alone to create a limited, high-specificity index of k-node induced subgraphs called k-graphlets (analogous to BLASTS's k-mers). The index is constructed so that, if significant common network topology exists between two graphs, their indexes are likely to overlap. BLANT-seed then queries the indexes of two networks to generate a list of common k-graphlets which, when paired, form a seed pair. Our companion paper (submitted elsewhere) describes BLANT-extend, which "grows" these seeds to larger local alignments, again using only topological information.
Long-term Reproducibility for Neural Architecture Search
Authors: David Towers, Matthew Forshaw, Amir Atapour-Abarghouei, Andrew Stephen McGough
Date: 2022-07-11 00:00:00
Tasks: Neural Architecture Search

It is a sad reflection of modern academia that code is often ignored after publication -- there is no academic 'kudos' for bug fixes / maintenance. Code is often unavailable or, if available, contains bugs, is incomplete, or relies on out-of-date / unavailable libraries. This has a significant impact on reproducibility and general scientific progress. Neural Architecture Search (NAS) is no exception to this, with some prior work in reproducibility. However, we argue that these do not consider long-term reproducibility issues. We therefore propose a checklist for long-term NAS reproducibility. We evaluate our checklist against common NAS approaches along with proposing how we can retrospectively make these approaches more long-term reproducible.
BLANT: Basic Local Alignment of Network Topology, Part 2: Topology-only Extension Beyond Graphlet Seeds
Authors: Tingyin Ding, Utsav Jain, Wayne B. Hayes
Date: 2022-07-10 00:00:00
Tasks: CPU

BLAST is a standard tool in bioinformatics for creating local sequence alignments using a "seed-and-extend" approach. Here we introduce an analogous seed-and-extend algorithm that produces local network alignments: BLANT (Basic Local Alignment of Network Topology). In Part 1, we introduced BLANT-seed, which generates graphlet-based seeds using only topological information. Here, in Part 2, we describe BLANT-extend, which "grows" seeds to larger local alignments using only topological information. We allow the user to specify bounds on several measures an alignment must satisfy, including the edge density, edge commonality (i.e., aligned edges), and node-pair similarity if such a measure is used; the latter allows the inclusion of sequence-based similarity, if desired, as well as local topological constraints. BLANT-extend is able to enumerate all possible alignments satisfying the bounds that can be grown from each seed, within a specified CPU time or number of generated alignments. While topology-driven local network alignment has a wide variety of potential applications outside bioinformatics, here we focus on the alignment of Protein-Protein Interaction (PPI) networks. We show that BLANT is capable of finding large, high-quality local alignments when the networks are known to have high topological similarity -- for example recovering hundreds of orthologs between networks of the recent Integrated Interaction Database (IID). Predictably, however, it performs less well when true topological similarity is absent, as is the case in most current experimental PPI networks that are noisy and have wide disparity in edge density which results in low common coverage.
Repairing Neural Networks by Leaving the Right Past Behind
Authors: Ryutaro Tanno, Melanie F. Pradier, Aditya Nori, Yingzhen Li
Date: 2022-07-11 00:00:00
Tasks: Continual Learning

Prediction failures of machine learning models often arise from deficiencies in training data, such as incorrect labels, outliers, and selection biases. However, such data points that are responsible for a given failure mode are generally not known a priori, let alone a mechanism for repairing the failure. This work draws on the Bayesian view of continual learning, and develops a generic framework for both, identifying training examples that have given rise to the target failure, and fixing the model through erasing information about them. This framework naturally allows leveraging recent advances in continual learning to this new problem of model repairment, while subsuming the existing works on influence functions and data deletion as specific instances. Experimentally, the proposed approach outperforms the baselines for both identification of detrimental training data and fixing model failures in a generalisable manner.
Matching Normalizing Flows and Probability Paths on Manifolds
Authors: Heli Ben-Hamu, samuel cohen, Joey Bose, Brandon Amos, Aditya Grover, Maximilian Nickel, Ricky T. Q. Chen, Yaron Lipman
Date: 2022-07-11 00:00:00

Continuous Normalizing Flows (CNFs) are a class of generative models that transform a prior distribution to a model distribution by solving an ordinary differential equation (ODE). We propose to train CNFs on manifolds by minimizing probability path divergence (PPD), a novel family of divergences between the probability density path generated by the CNF and a target probability density path. PPD is formulated using a logarithmic mass conservation formula which is a linear first order partial differential equation relating the log target probabilities and the CNF's defining vector field. PPD has several key benefits over existing methods: it sidesteps the need to solve an ODE per iteration, readily applies to manifold data, scales to high dimensions, and is compatible with a large family of target paths interpolating pure noise and data in finite time. Theoretically, PPD is shown to bound classical probability divergences. Empirically, we show that CNFs learned by minimizing PPD achieve state-of-the-art results in likelihoods and sample quality on existing low-dimensional manifold benchmarks, and is the first example of a generative model to scale to moderately high dimensional manifolds.
A Baselined Gated Attention Recurrent Network for Request Prediction in Ridesharing
Authors: Jingran Shen, Nikos Tziritas, Georgios Theodoropoulos
Date: 2022-07-11 00:00:00
Tasks: Prediction

Ridesharing has received global popularity due to its convenience and cost efficiency for both drivers and passengers and its strong potential to contribute to the implementation of the UN Sustainable Development Goals. As a result, recent years have witnessed an explosion of research interest in the RSODP (Origin-Destination Prediction for Ridesharing) problem with the goal of predicting the future ridesharing requests and providing schedules for vehicles ahead of time. Most of the existing prediction models utilise Deep Learning. However, they fail to effectively consider both spatial and temporal dynamics. In this paper the Baselined Gated Attention Recurrent Network (BGARN), is proposed, which uses graph convolution with multi-head gated attention to extract spatial features, a recurrent module to extract temporal features, and a baselined transferring layer to calculate the final results. The model is implemented with PyTorch and DGL (Deep Graph Library) and is experimentally evaluated using the New York Taxi Demand Dataset. The results show that BGARN outperforms all the other existing models in terms of prediction accuracy.
Learning Large-scale Universal User Representation with Sparse Mixture of Experts
Authors: Caigao Jiang, Siqiao Xue, James Zhang, Lingyue Liu, Zhibo Zhu, Hongyan Hao
Date: 2022-07-11 00:00:00
Tasks: Mixture-of-Experts

Learning user sequence behaviour embedding is very sophisticated and challenging due to the complicated feature interactions over time and high dimensions of user features. Recent emerging foundation models, e.g., BERT and its variants, encourage a large body of researchers to investigate in this field. However, unlike natural language processing (NLP) tasks, the parameters of user behaviour model come mostly from user embedding layer, which makes most existing works fail in training a universal user embedding of large scale. Furthermore, user representations are learned from multiple downstream tasks, and the past research work do not address the seesaw phenomenon. In this paper, we propose SUPERMOE, a generic framework to obtain high quality user representation from multiple tasks. Specifically, the user behaviour sequences are encoded by MoE transformer, and we can thus increase the model capacity to billions of parameters, or even to trillions of parameters. In order to deal with seesaw phenomenon when learning across multiple tasks, we design a new loss function with task indicators. We perform extensive offline experiments on public datasets and online experiments on private real-world business scenarios. Our approach achieves the best performance over state-of-the-art models, and the results demonstrate the effectiveness of our framework.
Partial Resampling of Imbalanced Data
Authors: Firuz Kamalov, Amir F. Atiya, Dina Elreedy
Date: 2022-07-11 00:00:00

Imbalanced data is a frequently encountered problem in machine learning. Despite a vast amount of literature on sampling techniques for imbalanced data, there is a limited number of studies that address the issue of the optimal sampling ratio. In this paper, we attempt to fill the gap in the literature by conducting a large scale study of the effects of sampling ratio on classification accuracy. We consider 10 popular sampling methods and evaluate their performance over a range of ratios based on 20 datasets. The results of the numerical experiments suggest that the optimal sampling ratio is between 0.7 and 0.8 albeit the exact ratio varies depending on the dataset. Furthermore, we find that while factors such the original imbalance ratio or the number of features do not play a discernible role in determining the optimal ratio, the number of samples in the dataset may have a tangible effect.
Mixed vine copula flows for flexible modelling of neural dependencies
Authors: Lazaros Mitskopoulos, Theoklitos Amvrosiadis, Arno Onken
Date: 2022-07-11 00:00:00

Recordings of complex neural population responses provide a unique opportunity for advancing our understanding of neural information processing at multiple scales and improving performance of brain computer interfaces. However, most existing analytical techniques fall short of capturing the complexity of interactions within the concerted population activity. Vine copula-based approaches have shown to be successful at addressing complex high-order dependencies within the population, disentangled from the single-neuron statistics. However, most applications have focused on parametric copulas which bear the risk of misspecifying dependence structures. In order to avoid this risk, we adopted a fully non-parametric approach for the single-neuron margins and copulas by using Neural Spline Flows (NSF). We validated the NSF framework on simulated data of continuous and discrete type with various forms of dependency structures and with different dimensionality. Overall, NSFs performed similarly to existing non-parametric estimators, while allowing for considerably faster and more flexible sampling which also enables faster Monte Carlo estimation of copula entropy. Moreover, our framework was able to capture low and higher order heavy tail dependencies in neuronal responses recorded in the mouse primary visual cortex during a visual learning task while the animal was navigating a virtual reality environment. These findings highlight an often ignored aspect of complexity in coordinated neuronal activity which can be important for understanding and deciphering collective neural dynamics for neurotechnological applications.
Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm
Authors: Lechao Xiao, Jeffrey Pennington
Date: 2022-07-11 00:00:00
Tasks: Open-Ended Question Answering, Triplet

Although learning in high dimensions is commonly believed to suffer from the curse of dimensionality, modern machine learning methods often exhibit an astonishing power to tackle a wide range of challenging real-world learning problems without using abundant amounts of data. How exactly these methods break this curse remains a fundamental open question in the theory of deep learning. While previous efforts have investigated this question by studying the data (D), model (M), and inference algorithm (I) as independent modules, in this paper, we analyze the triplet (D, M, I) as an integrated system and identify important synergies that help mitigate the curse of dimensionality. We first study the basic symmetries associated with various learning algorithms (M, I), focusing on four prototypical architectures in deep learning: fully-connected networks (FCN), locally-connected networks (LCN), and convolutional networks with and without pooling (GAP/VEC). We find that learning is most efficient when these symmetries are compatible with those of the data distribution and that performance significantly deteriorates when any member of the (D, M, I) triplet is inconsistent or suboptimal.
Multi-Study Boosting: Theoretical Considerations for Merging vs. Ensembling
Authors: Cathy Shyr, Pragya Sur, Giovanni Parmigiani, Prasad Patil
Date: 2022-07-11 00:00:00

Cross-study replicability is a powerful model evaluation criterion that emphasizes generalizability of predictions. When training cross-study replicable prediction models, it is critical to decide between merging and treating the studies separately. We study boosting algorithms in the presence of potential heterogeneity in predictor-outcome relationships across studies and compare two multi-study learning strategies: 1) merging all the studies and training a single model, and 2) multi-study ensembling, which involves training a separate model on each study and ensembling the resulting predictions. In the regression setting, we provide theoretical guidelines based on an analytical transition point to determine whether it is more beneficial to merge or to ensemble for boosting with linear learners. In addition, we characterize a bias-variance decomposition of estimation error for boosting with component-wise linear learners. We verify the theoretical transition point result in simulation and illustrate how it can guide the decision on merging vs. ensembling in an application to breast cancer gene expression data.
A multi-level interpretable sleep stage scoring system by infusing experts' knowledge into a deep network architecture
Authors: Hamid Niknazar, Sara C. Mednick
Date: 2022-07-11 00:00:00
Tasks: Decision Making, EEG, Electroencephalogram (EEG), Time Series, Time Series Analysis, Time Series Classification

In recent years, deep learning has shown potential and efficiency in a wide area including computer vision, image and signal processing. Yet, translational challenges remain for user applications due to a lack of interpretability of algorithmic decisions and results. This black box problem is particularly problematic for high-risk applications such as medical-related decision-making. The current study goal was to design an interpretable deep learning system for time series classification of electroencephalogram (EEG) for sleep stage scoring as a step toward designing a transparent system. We have developed an interpretable deep neural network that includes a kernel-based layer based on a set of principles used for sleep scoring by human experts in the visual analysis of polysomnographic records. A kernel-based convolutional layer was defined and used as the first layer of the system and made available for user interpretation. The trained system and its results were interpreted in four levels from the microstructure of EEG signals, such as trained kernels and the effect of each kernel on the detected stages, to macrostructures, such as the transition between stages. The proposed system demonstrated greater performance than prior studies and the results of interpretation showed that the system learned information which was consistent with expert knowledge.
Domain Confused Contrastive Learning for Unsupervised Domain Adaptation
Authors: Quanyu Long, Tianze Luo, Wenya Wang, Sinno Jialin Pan
Date: 2022-07-10 00:00:00
Proceeding: NAACL 2022 7
Tasks: Contrastive Learning, Domain Adaptation, Unsupervised Domain Adaptation

In this work, we study Unsupervised Domain Adaptation (UDA) in a challenging self-supervised approach. One of the difficulties is how to learn task discrimination in the absence of target labels. Unlike previous literature which directly aligns cross-domain distributions or leverages reverse gradient, we propose Domain Confused Contrastive Learning (DCCL) to bridge the source and the target domains via domain puzzles, and retain discriminative representations after adaptation. Technically, DCCL searches for a most domain-challenging direction and exquisitely crafts domain confused augmentations as positive pairs, then it contrastively encourages the model to pull representations towards the other domain, thus learning more stable and effective domain invariances. We also investigate whether contrastive learning necessarily helps with UDA when performing other data augmentations. Extensive experiments demonstrate that DCCL significantly outperforms baselines.
Multi-task Envisioning Transformer-based Autoencoder for Corporate Credit Rating Migration Early Prediction
Authors: Han Yue, Steve Xia, Hongfu Liu
Date: 2022-07-10 00:00:00
Tasks: Decision Making, Prediction

Corporate credit ratings issued by third-party rating agencies are quantified assessments of a company's creditworthiness. Credit Ratings highly correlate to the likelihood of a company defaulting on its debt obligations. These ratings play critical roles in investment decision-making as one of the key risk factors. They are also central to the regulatory framework such as BASEL II in calculating necessary capital for financial institutions. Being able to predict rating changes will greatly benefit both investors and regulators alike. In this paper, we consider the corporate credit rating migration early prediction problem, which predicts the credit rating of an issuer will be upgraded, unchanged, or downgraded after 12 months based on its latest financial reporting information at the time. We investigate the effectiveness of different standard machine learning algorithms and conclude these models deliver inferior performance. As part of our contribution, we propose a new Multi-task Envisioning Transformer-based Autoencoder (META) model to tackle this challenging problem. META consists of Positional Encoding, Transformer-based Autoencoder, and Multi-task Prediction to learn effective representations for both migration prediction and rating prediction. This enables META to better explore the historical data in the training stage for one-year later prediction. Experimental results show that META outperforms all baseline models.
Automatic differentiation and the optimization of differential equation models in biology
Authors: Steven A. Frank
Date: 2022-07-10 00:00:00

A computational revolution unleashed the power of artificial neural networks. At the heart of that revolution is automatic differentiation, which calculates the derivative of a performance measure relative to a large number of parameters. Differentiation enhances the discovery of improved performance in large models, an achievement that was previously difficult or impossible. Recently, a second computational advance optimizes the temporal trajectories traced by differential equations. Optimization requires differentiating a measure of performance over a trajectory, such as the closeness of tracking the environment, with respect to the parameters of the differential equations. Because model trajectories are usually calculated numerically by multistep algorithms, such as Runge-Kutta, the automatic differentiation must be passed through the numerical algorithm. This article explains how such automatic differentiation of trajectories is achieved. It also discusses why such computational breakthroughs are likely to advance theoretical and statistical studies of biological problems, in which one can consider variables as dynamic paths over time and space. Many common problems arise between improving success in computational learning models over performance landscapes, improving evolutionary fitness over adaptive landscapes, and improving statistical fits to data over information landscapes.