pwc-archive/papers-with-abstracts huggingface dataset:
Deconfounded Causal Collaborative Filtering
Authors: Shuyuan Xu, Juntao Tan, Shelby Heinecke, Jia Li, Yongfeng Zhang
Date: 2021-10-14 00:00:00
Tasks: Collaborative Filtering, Recommendation Systems

Recommender systems may be confounded by various types of confounding factors (also called confounders) that may lead to inaccurate recommendations and sacrificed recommendation performance. Current approaches to solving the problem usually design each specific model for each specific confounder. However, real-world systems may include a huge number of confounders and thus designing each specific model for each specific confounder could be unrealistic. More importantly, except for those ``explicit confounders'' that experts can manually identify and process such as item's position in the ranking list, there are also many ``latent confounders'' that are beyond the imagination of experts. For example, users' rating on a song may depend on their current mood or the current weather, and users' preference on ice creams may depend on the air temperature. Such latent confounders may be unobservable in the recorded training data. To solve the problem, we propose Deconfounded Causal Collaborative Filtering (DCCF). We first frame user behaviors with unobserved confounders into a causal graph, and then we design a front-door adjustment model carefully fused with machine learning to deconfound the influence of unobserved confounders. Experiments on real-world datasets show that our method is able to deconfound unobserved confounders to achieve better recommendation performance.
A Cubic Regularization Approach for Finding Local Minimax Points in Nonconvex Minimax Optimization
Authors: Ziyi Chen, Zhengyang Hu, Qunwei Li, Zhe Wang, Yi Zhou
Date: 2021-10-14 00:00:00

Gradient descent-ascent (GDA) is a widely used algorithm for minimax optimization. However, GDA has been proved to converge to stationary points for nonconvex minimax optimization, which are suboptimal compared with local minimax points. In this work, we develop cubic regularization (CR) type algorithms that globally converge to local minimax points in nonconvex-strongly-concave minimax optimization. We first show that local minimax points are equivalent to second-order stationary points of a certain envelope function. Then, inspired by the classic cubic regularization algorithm, we propose an algorithm named Cubic-LocalMinimax for finding local minimax points, and provide a comprehensive convergence analysis by leveraging its intrinsic potential function. Specifically, we establish the global convergence of Cubic-LocalMinimax to a local minimax point at a sublinear convergence rate and characterize its iteration complexity. Also, we propose a GDA-based solver for solving the cubic subproblem involved in Cubic-LocalMinimax up to certain pre-defined accuracy, and analyze the overall gradient and Hessian-vector product computation complexities of such an inexact Cubic-LocalMinimax algorithm. Moreover, we propose a stochastic variant of Cubic-LocalMinimax for large-scale minimax optimization, and characterize its sample complexity under stochastic sub-sampling. Experimental results demonstrate faster convergence of our stochastic Cubic-LocalMinimax than some existing algorithms.
CloudPred: Predicting Patient Phenotypes From Single-cell RNA-seq
Authors: Bryan He, Matthew Thomson, Meena Subramaniam, Richard Perez, Chun Jimmie Ye, James Zou
Date: 2021-10-13 00:00:00
Tasks: Interpretable Machine Learning, Prognosis

Single-cell RNA sequencing (scRNA-seq) has the potential to provide powerful, high-resolution signatures to inform disease prognosis and precision medicine. This paper takes an important first step towards this goal by developing an interpretable machine learning algorithm, CloudPred, to predict individuals' disease phenotypes from their scRNA-seq data. Predicting phenotype from scRNA-seq is challenging for standard machine learning methods -- the number of cells measured can vary by orders of magnitude across individuals and the cell populations are also highly heterogeneous. Typical analysis creates pseudo-bulk samples which are biased toward prior annotations and also lose the single cell resolution. CloudPred addresses these challenges via a novel end-to-end differentiable learning algorithm which is coupled with a biologically informed mixture of cell types model. CloudPred automatically infers the cell subpopulation that are salient for the phenotype without prior annotations. We developed a systematic simulation platform to evaluate the performance of CloudPred and several alternative methods we propose, and find that CloudPred outperforms the alternative methods across several settings. We further validated CloudPred on a real scRNA-seq dataset of 142 lupus patients and controls. CloudPred achieves AUROC of 0.98 while identifying a specific subpopulation of CD4 T cells whose presence is highly indicative of lupus. CloudPred is a powerful new framework to predict clinical phenotypes from scRNA-seq data and to identify relevant cells.
Variance Minimization in the Wasserstein Space for Invariant Causal Prediction
Authors: Guillaume Martinet, Alexander Strzalkowski, Barbara E. Engelhardt
Date: 2021-10-13 00:00:00
Tasks: Causal Discovery, Causal Inference

Selecting powerful predictors for an outcome is a cornerstone task for machine learning. However, some types of questions can only be answered by identifying the predictors that causally affect the outcome. A recent approach to this causal inference problem leverages the invariance property of a causal mechanism across differing experimental environments (Peters et al., 2016; Heinze-Deml et al., 2018). This method, invariant causal prediction (ICP), has a substantial computational defect -- the runtime scales exponentially with the number of possible causal variables. In this work, we show that the approach taken in ICP may be reformulated as a series of nonparametric tests that scales linearly in the number of predictors. Each of these tests relies on the minimization of a novel loss function -- the Wasserstein variance -- that is derived from tools in optimal transport theory and is used to quantify distributional variability across environments. We prove under mild assumptions that our method is able to recover the set of identifiable direct causes, and we demonstrate in our experiments that it is competitive with other benchmark causal discovery algorithms.
Robust MIMO Detection using Hypernetworks with Learned Regularizers
Authors: Nicolas Zilberstein, Chris Dick, Rahman Doost-Mohammady, Ashutosh Sabharwal, Santiago Segarra
Date: 2021-10-13 00:00:00

Optimal symbol detection in multiple-input multiple-output (MIMO) systems is known to be an NP-hard problem. Recently, there has been a growing interest to get reasonably close to the optimal solution using neural networks while keeping the computational complexity in check. However, existing work based on deep learning shows that it is difficult to design a generic network that works well for a variety of channels. In this work, we propose a method that tries to strike a balance between symbol error rate (SER) performance and generality of channels. Our method is based on hypernetworks that generate the parameters of a neural network-based detector that works well on a specific channel. We propose a general framework by regularizing the training of the hypernetwork with some pre-trained instances of the channel-specific method. Through numerical experiments, we show that our proposed method yields high performance for a set of prespecified channel realizations while generalizing well to all channels drawn from a specific distribution.
Deep Metric Learning with Locality Sensitive Angular Loss for Self-Correcting Source Separation of Neural Spiking Signals
Authors: Alexander Kenneth Clarke, Dario Farina
Date: 2021-10-13 00:00:00
Tasks: Metric Learning, Time Series, Time Series Analysis

Neurophysiological time series, such as electromyographic signal and intracortical recordings, are typically composed of many individual spiking sources, the recovery of which can give fundamental insights into the biological system of interest or provide neural information for man-machine interfaces. For this reason, source separation algorithms have become an increasingly important tool in neuroscience and neuroengineering. However, in noisy or highly multivariate recordings these decomposition techniques often make a large number of errors, which degrades human-machine interfacing applications and often requires costly post-hoc manual cleaning of the output label set of spike timestamps. To address both the need for automated post-hoc cleaning and robust separation filters we propose a methodology based on deep metric learning, using a novel loss function which maintains intra-class variance, creating a rich embedding space suitable for both label cleaning and the discovery of new activations. We then validate this method with an artificially corrupted label set based on source-separated high-density surface electromyography recordings, recovering the original timestamps even in extreme degrees of feature and class-dependent label noise. This approach enables a neural network to learn to accurately decode neurophysiological time series using any imperfect method of labelling the signal.
Why Out-of-distribution Detection in CNNs Does Not Like Mahalanobis -- and What to Use Instead
Authors: Kamil Szyc, Tomasz Walkowiak, Henryk Maciejewski
Date: 2021-10-13 00:00:00
Tasks: Out-of-Distribution Detection, Playing the Game of 2048

Convolutional neural networks applied for real-world classification tasks need to recognize inputs that are far or out-of-distribution (OoD) with respect to the known or training data. To achieve this, many methods estimate class-conditional posterior probabilities and use confidence scores obtained from the posterior distributions. Recent works propose to use multivariate Gaussian distributions as models of posterior distributions at different layers of the CNN (i.e., for low- and upper-level features), which leads to the confidence scores based on the Mahalanobis distance. However, this procedure involves estimating probability density in high dimensional data using the insufficient number of observations (e.g. the dimensionality of features at the last two layers in the ResNet-101 model are 2048 and 1024, with ca. 1000 observations per class used to estimate density). In this work, we want to address this problem. We show that in many OoD studies in high-dimensional data, LOF-based (Local Outlierness-Factor) methods outperform the parametric, Mahalanobis distance-based methods. This motivates us to propose the nonparametric, LOF-based method of generating the confidence scores for CNNs. We performed several feasibility studies involving ResNet-101 and EffcientNet-B3, based on CIFAR-10 and ImageNet (as known data), and CIFAR-100, SVHN, ImageNet2010, Places365, or ImageNet-O (as outliers). We demonstrated that nonparametric LOF-based confidence estimation can improve current Mahalanobis-based SOTA or obtain similar performance in a simpler way.
Bond Default Prediction with Text Embeddings, Undersampling and Deep Learning
Authors: Luke Jordan
Date: 2021-10-13 00:00:00
Tasks: Deep Learning

The special and important problems of default prediction for municipal bonds are addressed using a combination of text embeddings from a pre-trained transformer network, a fully connected neural network, and synthetic oversampling. The combination of these techniques provides significant improvement in performance over human estimates, linear models, and boosted ensemble models, on data with extreme imbalance. Less than 0.2% of municipal bonds default, but our technique predicts 9 out of 10 defaults at the time of issue, without using bond ratings, at a cost of false positives on less than 0.1% non-defaulting bonds. The results hold the promise of reducing the cost of capital for local public goods, which are vital for society, and bring techniques previously used in personal credit and public equities (or national fixed income), as well as the current generation of embedding techniques, to sub-sovereign credit decisions.
On the Convergence Theory for Hessian-Free Bilevel Algorithms
Authors: Daouda Sow, Kaiyi Ji, Yingbin Liang
Date: 2021-10-13 00:00:00
Tasks: Bilevel Optimization, Meta-Learning

Bilevel optimization has arisen as a powerful tool in modern machine learning. However, due to the nested structure of bilevel optimization, even gradient-based methods require second-order derivative approximations via Jacobian- or/and Hessian-vector computations, which can be costly and unscalable in practice. Recently, Hessian-free bilevel schemes have been proposed to resolve this issue, where the general idea is to use zeroth- or first-order methods to approximate the full hypergradient of the bilevel problem. However, we empirically observe that such approximation can lead to large variance and unstable training, but estimating only the response Jacobian matrix as a partial component of the hypergradient turns out to be extremely effective. To this end, we propose a new Hessian-free method, which adopts the zeroth-order-like method to approximate the response Jacobian matrix via taking difference between two optimization paths. Theoretically, we provide the convergence rate analysis for the proposed algorithms, where our key challenge is to characterize the approximation and smoothness properties of the trajectory-dependent estimator, which can be of independent interest. This is the first known convergence rate result for this type of Hessian-free bilevel algorithms. Experimentally, we demonstrate that the proposed algorithms outperform baseline bilevel optimizers on various bilevel problems. Particularly, in our experiment on few-shot meta-learning with ResNet-12 network over the miniImageNet dataset, we show that our algorithm outperforms baseline meta-learning algorithms, while other baseline bilevel optimizers do not solve such meta-learning problems within a comparable time frame.
Sustainability Through Cognition Aware Safety Systems -- Next Level Human-Machine-Interaction
Authors: Juergen Mangler, Konrad Diwol, Dieter Etz, Stefanie Rinderle-Ma, Alois Ferscha, Gerald Reiner, Wolfgang Kastner, Sebastien Bougain, Christoph Pollak, Michael Haslgrübler
Date: 2021-10-13 00:00:00

Industrial Safety deals with the physical integrity of humans, machines and the environment when they interact during production scenarios. Industrial Safety is subject to a rigorous certification process that leads to inflexible settings, in which all changes are forbidden. With the progressing introduction of smart robotics and smart machinery to the factory floor, combined with an increasing shortage of skilled workers, it becomes imperative that safety scenarios incorporate a flexible handling of the boundary between humans, machines and the environment. In order to increase the well-being of workers, reduce accidents, and compensate for different skill sets, the configuration of machines and the factory floor should be dynamically adapted, while still enforcing functional safety requirements. The contribution of this paper is as follows: (1) We present a set of three scenarios, and discuss how industrial safety mechanisms could be augmented through dynamic changes to the work environment in order to decrease potential accidents, and thus increase productivity. (2) We introduce the concept of a Cognition Aware Safety System (CASS) and its architecture. The idea behind CASS is to integrate AI based reasoning about human load, stress, and attention with AI based selection of actions to avoid the triggering of safety stops. (3) And finally, we will describe the required performance measurement dimensions for a quantitative performance measurement model to enable a comprehensive (triple bottom line) impact assessment of CASS. Additionally we introduce a detailed guideline for expert interviews to explore the feasibility of the approach for given scenarios.
WAFFLE: Weighted Averaging for Personalized Federated Learning
Authors: Martin Beaussart, Felix Grimberg, Mary-Anne Hartley, Martin Jaggi
Date: 2021-10-13 00:00:00
Tasks: Federated Learning, Personalized Federated Learning

In federated learning, model personalization can be a very effective strategy to deal with heterogeneous training data across clients. We introduce WAFFLE (Weighted Averaging For Federated LEarning), a personalized collaborative machine learning algorithm that leverages stochastic control variates for faster convergence. WAFFLE uses the Euclidean distance between clients' updates to weigh their individual contributions and thus minimize the personalized model loss on the specific agent of interest. Through a series of experiments, we compare our new approach to two recent personalized federated learning methods--Weight Erosion and APFL--as well as two general FL methods--Federated Averaging and SCAFFOLD. Performance is evaluated using two categories of non-identical client data distributions--concept shift and label skew--on two image data sets (MNIST and CIFAR10). Our experiments demonstrate the comparative effectiveness of WAFFLE, as it achieves or improves accuracy with faster convergence.
Language Modelling via Learning to Rank
Authors: Arvid Frydenlund, Gagandeep Singh, Frank Rudzicz
Date: 2021-10-13 00:00:00
Tasks: Knowledge Distillation, Language Modelling, Learning-To-Rank, Structured Prediction

We consider language modelling (LM) as a multi-label structured prediction task by re-framing training from solely predicting a single ground-truth word to ranking a set of words which could continue a given context. To avoid annotating top-$k$ ranks, we generate them using pre-trained LMs: GPT-2, BERT, and Born-Again models. This leads to a rank-based form of knowledge distillation (KD). We also develop a method using $N$-grams to create a non-probabilistic teacher which generates the ranks without the need of a pre-trained LM. We confirm the hypotheses that we can treat LMing as a ranking task and that we can do so without the use of a pre-trained LM. We show that rank-based KD generally improves perplexity (PPL), often with statistical significance, when compared to Kullback-Leibler-based KD. Surprisingly, given the simplicity of the method, $N$-grams act as competitive teachers and achieve similar performance as using either BERT or a Born-Again model teachers. GPT-2 always acts as the best teacher, though, and using it and a Transformer-XL student on Wiki-02, rank-based KD reduces a cross-entropy baseline from 65.27 to 55.94 and against a KL-based KD of 56.70.
Challenges for Unsupervised Anomaly Detection in Particle Physics
Authors: Katherine Fraser, Samuel Homiller, Rashmish K. Mishra, Bryan Ostdiek, Matthew D. Schwartz
Date: 2021-10-13 00:00:00
Tasks: Anomaly Detection, Unsupervised Anomaly Detection

Anomaly detection relies on designing a score to determine whether a particular event is uncharacteristic of a given background distribution. One way to define a score is to use autoencoders, which rely on the ability to reconstruct certain types of data (background) but not others (signals). In this paper, we study some challenges associated with variational autoencoders, such as the dependence on hyperparameters and the metric used, in the context of anomalous signal (top and $W$) jets in a QCD background. We find that the hyperparameter choices strongly affect the network performance and that the optimal parameters for one signal are non-optimal for another. In exploring the networks, we uncover a connection between the latent space of a variational autoencoder trained using mean-squared-error and the optimal transport distances within the dataset. We then show that optimal transport distances to representative events in the background dataset can be used directly for anomaly detection, with performance comparable to the autoencoders. Whether using autoencoders or optimal transport distances for anomaly detection, we find that the choices that best represent the background are not necessarily best for signal identification. These challenges with unsupervised anomaly detection bolster the case for additional exploration of semi-supervised or alternative approaches.
Style-based quantum generative adversarial networks for Monte Carlo events
Authors: Carlos Bravo-Prieto, Julien Baglio, Marco Cè, Anthony Francis, Dorota M. Grabowska, Stefano Carrazza
Date: 2021-10-13 00:00:00
Tasks: Data Augmentation

We propose and assess an alternative quantum generator architecture in the context of generative adversarial learning for Monte Carlo event generation, used to simulate particle physics processes at the Large Hadron Collider (LHC). We validate this methodology by implementing the quantum network on artificial data generated from known underlying distributions. The network is then applied to Monte Carlo-generated datasets of specific LHC scattering processes. The new quantum generator architecture leads to a generalization of the state-of-the-art implementations, achieving smaller Kullback-Leibler divergences even with shallow-depth networks. Moreover, the quantum generator successfully learns the underlying distribution functions even if trained with small training sample sets; this is particularly interesting for data augmentation applications. We deploy this novel methodology on two different quantum hardware architectures, trapped-ion and superconducting technologies, to test its hardware-independent viability.
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
Authors: Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, Jie Tang
Date: 2021-10-14 00:00:00
Tasks: Language Modeling, Language Modelling

Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is an implementation of Deep Prompt Tuning \cite{li2021prefix,qin2021learning} optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research.Our code and data are released at https://github.com/THUDM/P-tuning-v2.
Can Explanations Be Useful for Calibrating Black Box Models?
Authors: Xi Ye, Greg Durrett
Date: 2021-10-14 00:00:00
Proceeding: ACL 2022 5
Tasks: Extractive Question-Answering, Few-Shot Learning, Natural Language Inference, Question Answering

NLP practitioners often want to take existing trained models and apply them to data from new domains. While fine-tuning or few-shot learning can be used to adapt a base model, there is no single recipe for making these techniques work; moreover, one may not have access to the original model weights if it is deployed as a black box. We study how to improve a black box model's performance on a new domain by leveraging explanations of the model's behavior. Our approach first extracts a set of features combining human intuition about the task with model attributions generated by black box interpretation techniques, then uses a simple calibrator, in the form of a classifier, to predict whether the base model was correct or not. We experiment with our method on two tasks, extractive question answering and natural language inference, covering adaptation from several pairs of domains with limited target-domain data. The experimental results across all the domain pairs show that explanations are useful for calibrating these models, boosting accuracy when predictions do not have to be returned on every example. We further show that the calibration model transfers to some extent between tasks.