pwc-archive/papers-with-abstracts huggingface dataset:
Adversarial Self-Attention for Language Understanding
Authors: Hongqiu Wu, Ruixue Ding, Hai Zhao, Pengjun Xie, Fei Huang, Min Zhang
Date: 2022-06-25 00:00:00
Tasks: Machine Reading Comprehension, Named Entity Recognition (NER), Natural Language Inference, Paraphrase Identification, Semantic Similarity, Semantic Textual Similarity, Sentiment Analysis

Deep neural models (e.g. Transformer) naturally learn spurious features, which create a ``shortcut'' between the labels and inputs, thus impairing the generalization and robustness. This paper advances the self-attention mechanism to its robust variant for Transformer-based pre-trained language models (e.g. BERT). We propose \textit{Adversarial Self-Attention} mechanism (ASA), which adversarially biases the attentions to effectively suppress the model reliance on features (e.g. specific keywords) and encourage its exploration of broader semantics. We conduct a comprehensive evaluation across a wide range of tasks for both pre-training and fine-tuning stages. For pre-training, ASA unfolds remarkable performance gains compared to naive training for longer steps. For fine-tuning, ASA-empowered models outweigh naive models by a large margin considering both generalization and robustness.
Mitigating sampling bias in risk-based active learning via an EM algorithm
Authors: Aidan J. Hughes, Lawrence A. Bull, Paul Gardner, Nikolaos Dervilis, Keith Worden
Date: 2022-06-25 00:00:00
Tasks: Active Learning, Decision Making

Risk-based active learning is an approach to developing statistical classifiers for online decision-support. In this approach, data-label querying is guided according to the expected value of perfect information for incipient data points. For SHM applications, the value of information is evaluated with respect to a maintenance decision process, and the data-label querying corresponds to the inspection of a structure to determine its health state. Sampling bias is a known issue within active-learning paradigms; this occurs when an active learning process over- or undersamples specific regions of a feature-space, thereby resulting in a training set that is not representative of the underlying distribution. This bias ultimately degrades decision-making performance, and as a consequence, results in unnecessary costs incurred. The current paper outlines a risk-based approach to active learning that utilises a semi-supervised Gaussian mixture model. The semi-supervised approach counteracts sampling bias by incorporating pseudo-labels for unlabelled data via an EM algorithm. The approach is demonstrated on a numerical example representative of the decision processes found in SHM.
Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernels
Authors: Mohamad Amin Mohamadi, Wonho Bae, Danica J. Sutherland
Date: 2022-06-25 00:00:00
Tasks: Active Learning

We propose a new method for approximating active learning acquisition strategies that are based on retraining with hypothetically-labeled candidate data points. Although this is usually infeasible with deep networks, we use the neural tangent kernel to approximate the result of retraining, and prove that this approximation works asymptotically even in an active learning setup -- approximating "look-ahead" selection criteria with far less computation required. This also enables us to conduct sequential active learning, i.e. updating the model in a streaming regime, without needing to retrain the model with SGD after adding each new data point. Moreover, our querying strategy, which better understands how the model's predictions will change by adding new data points in comparison to the standard ("myopic") criteria, beats other look-ahead strategies by large margins, and achieves equal or better performance compared to state-of-the-art methods on several benchmark datasets in pool-based active learning.
Generating Diverse Vocal Bursts with StyleGAN2 and MEL-Spectrograms
Authors: Marco Jiralerspong, Gauthier Gidel
Date: 2022-06-25 00:00:00
Tasks: FAD

We describe our approach for the generative emotional vocal burst task (ExVo Generate) of the ICML Expressive Vocalizations Competition. We train a conditional StyleGAN2 architecture on mel-spectrograms of preprocessed versions of the audio samples. The mel-spectrograms generated by the model are then inverted back to the audio domain. As a result, our generated samples substantially improve upon the baseline provided by the competition from a qualitative and quantitative perspective for all emotions. More precisely, even for our worst-performing emotion (awe), we obtain an FAD of 1.76 compared to the baseline of 4.81 (as a reference, the FAD between the train/validation sets for awe is 0.776).
PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance
Authors: Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, Tuo Zhao
Date: 2022-06-25 00:00:00
Tasks: image-classification, Image Classification, Natural Language Understanding, Question Answering

Large Transformer-based models have exhibited superior performance in various natural language processing and computer vision tasks. However, these models contain enormous amounts of parameters, which restrict their deployment to real-world applications. To reduce the model size, researchers prune these models based on the weights' importance scores. However, such scores are usually estimated on mini-batches during training, which incurs large variability/uncertainty due to mini-batch sampling and complicated training dynamics. As a result, some crucial weights could be pruned by commonly used pruning methods because of such uncertainty, which makes training unstable and hurts generalization. To resolve this issue, we propose PLATON, which captures the uncertainty of importance scores by upper confidence bound (UCB) of importance estimation. In particular, for the weights with low importance scores but high uncertainty, PLATON tends to retain them and explores their capacity. We conduct extensive experiments with several Transformer-based models on natural language understanding, question answering and image classification to validate the effectiveness of PLATON. Results demonstrate that PLATON manifests notable improvement under different sparsity levels. Our code is publicly available at https://github.com/QingruZhang/PLATON.
Integrating Machine Learning with Discrete Event Simulation for Improving Health Referral Processing in a Care Management Setting
Authors: Mohammed Mahyoub
Date: 2022-06-25 00:00:00
Tasks: Management, Prediction

Post-discharge care management coordinates patients' referrals to improve their health after being discharged from hospitals, especially elderly and chronically ill patients. In a care management setting, health referrals are processed by a specialized unit in the managed care organization (MCO), which interacts with many other entities including inpatient hospitals, insurance companies, and post-discharge care providers. In this paper, a machine-learning-guided discrete event simulation framework to improve health referrals processing is proposed. Random-forest-based prediction models are developed to predict the LOS and referral type. Two simulation models are constructed to represent the as-is configuration of the referral processing system and the intelligent system after incorporating the prediction functionality, respectively. By incorporating a prediction module for the referral processing system to plan and prioritize referrals, the overall performance was enhanced in terms of reducing the average referral creation delay time. This research will emphasize the role of post-discharge care management in improving health quality and reducing associated costs. Also, the paper demonstrates how to use integrated systems engineering methods for process improvement of complex healthcare systems.
Geometry Contrastive Learning on Heterogeneous Graphs
Authors: Shichao Zhu, Chuan Zhou, Anfeng Cheng, Shirui Pan, Shuaiqiang Wang, Dawei Yin, Bin Wang
Date: 2022-06-25 00:00:00
Tasks: Contrastive Learning, Node Classification, Node Clustering, Representation Learning, Self-Supervised Learning

Self-supervised learning (especially contrastive learning) methods on heterogeneous graphs can effectively get rid of the dependence on supervisory data. Meanwhile, most existing representation learning methods embed the heterogeneous graphs into a single geometric space, either Euclidean or hyperbolic. This kind of single geometric view is usually not enough to observe the complete picture of heterogeneous graphs due to their rich semantics and complex structures. Under these observations, this paper proposes a novel self-supervised learning method, termed as Geometry Contrastive Learning (GCL), to better represent the heterogeneous graphs when supervisory data is unavailable. GCL views a heterogeneous graph from Euclidean and hyperbolic perspective simultaneously, aiming to make a strong merger of the ability of modeling rich semantics and complex structures, which is expected to bring in more benefits for downstream tasks. GCL maximizes the mutual information between two geometric views by contrasting representations at both local-local and local-global semantic levels. Extensive experiments on four benchmarks data sets show that the proposed approach outperforms the strong baselines, including both unsupervised methods and supervised methods, on three tasks, including node classification, node clustering and similarity search.
A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel
Authors: Mohamad Amin Mohamadi, Wonho Bae, Danica J. Sutherland
Date: 2022-06-25 00:00:00

Empirical neural tangent kernels (eNTKs) can provide a good understanding of a given network's representation: they are often far less expensive to compute and applicable more broadly than infinite width NTKs. For networks with O output units (e.g. an O-class classifier), however, the eNTK on N inputs is of size $NO \times NO$, taking $O((NO)^2)$ memory and up to $O((NO)^3)$ computation. Most existing applications have therefore used one of a handful of approximations yielding $N \times N$ kernel matrices, saving orders of magnitude of computation, but with limited to no justification. We prove that one such approximation, which we call "sum of logits", converges to the true eNTK at initialization for any network with a wide final "readout" layer. Our experiments demonstrate the quality of this approximation for various uses across a range of settings.
Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning
Authors: Yang Yue, Bingyi Kang, Zhongwen Xu, Gao Huang, Shuicheng Yan
Date: 2022-06-25 00:00:00
Tasks: Contrastive Learning, Data Augmentation, Decision Making, Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Representation Learning, Value prediction

Deep reinforcement learning (RL) algorithms suffer severe performance degradation when the interaction data is scarce, which limits their real-world application. Recently, visual representation learning has been shown to be effective and promising for boosting sample efficiency in RL. These methods usually rely on contrastive learning and data augmentation to train a transition model for state prediction, which is different from how the model is used in RL--performing value-based planning. Accordingly, the learned representation by these visual methods may be good for recognition but not optimal for estimating state value and solving the decision problem. To address this issue, we propose a novel method, called value-consistent representation learning (VCR), to learn representations that are directly related to decision-making. More specifically, VCR trains a model to predict the future state (also referred to as the ''imagined state'') based on the current one and a sequence of actions. Instead of aligning this imagined state with a real state returned by the environment, VCR applies a $Q$-value head on both states and obtains two distributions of action values. Then a distance is computed and minimized to force the imagined state to produce a similar action value prediction as that by the real state. We develop two implementations of the above idea for the discrete and continuous action spaces respectively. We conduct experiments on Atari 100K and DeepMind Control Suite benchmarks to validate their effectiveness for improving sample efficiency. It has been demonstrated that our methods achieve new state-of-the-art performance for search-free RL algorithms.
Visual Auditor: Interactive Visualization for Detection and Summarization of Model Biases
Authors: David Munechika, Zijie J. Wang, Jack Reidy, Josh Rubin, Krishna Gade, Krishnaram Kenthapadi, Duen Horng Chau
Date: 2022-06-25 00:00:00

As machine learning (ML) systems become increasingly widespread, it is necessary to audit these systems for biases prior to their deployment. Recent research has developed algorithms for effectively identifying intersectional bias in the form of interpretable, underperforming subsets (or slices) of the data. However, these solutions and their insights are limited without a tool for visually understanding and interacting with the results of these algorithms. We propose Visual Auditor, an interactive visualization tool for auditing and summarizing model biases. Visual Auditor assists model validation by providing an interpretable overview of intersectional bias (bias that is present when examining populations defined by multiple features), details about relationships between problematic data slices, and a comparison between underperforming and overperforming data slices in a model. Our open-source tool runs directly in both computational notebooks and web browsers, making model auditing accessible and easily integrated into current ML development workflows. An observational user study in collaboration with domain experts at Fiddler AI highlights that our tool can help ML practitioners identify and understand model biases.
The Amenability Framework: Rethinking Causal Ordering Without Estimating Causal Effects
Authors: Carlos Fernández-Loría, Jorge Loría
Date: 2022-06-25 00:00:00
Tasks: Causal Inference, Decision Making, Learning-To-Rank

Who should we prioritize for intervention when we cannot estimate intervention effects? In many applied domains -- such as advertising, customer retention, and behavioral nudging -- prioritization is guided by predictive models that estimate outcome probabilities rather than causal effects. This paper investigates when these predictions (scores) can effectively rank individuals by their intervention effects, particularly when direct effect estimation is infeasible or unreliable. We propose a conceptual framework based on amenability -- an individual's latent proclivity to be influenced by an intervention -- and formalize conditions under which predictive scores serve as effective proxies for amenability. These conditions justify using non-causal scores for intervention prioritization, even when the scores do not directly estimate effects. We further show that, under plausible assumptions, predictive models can outperform causal effect estimators in ranking individuals by intervention effects. Empirical evidence from an advertising context supports our theoretical findings, demonstrating that predictive modeling can offer a more robust approach to targeting than effect estimation. Our framework suggests a shift in focus -- from estimating effects to inferring who is amenable -- as a practical and theoretically grounded strategy for prioritizing interventions in resource-constrained environments.
Infinite Impulse Response Graph Neural Networks for Cyberattack Localization in Smart Grids
Authors: Osman Boyaci, M. Rasoul Narimani, Katherine Davis, Erchin Serpedin
Date: 2022-06-25 00:00:00

This study employs Infinite Impulse Response (IIR) Graph Neural Networks (GNN) to efficiently model the inherent graph network structure of the smart grid data to address the cyberattack localization problem. First, we numerically analyze the empirical frequency response of the Finite Impulse Response (FIR) and IIR graph filters (GFs) to approximate an ideal spectral response. We show that, for the same filter order, IIR GFs provide a better approximation to the desired spectral response and they also present the same level of approximation to a lower order GF due to their rational type filter response. Second, we propose an IIR GNN model to efficiently predict the presence of cyberattacks at the bus level. Finally, we evaluate the model under various cyberattacks at both sample-wise (SW) and bus-wise (BW) level, and compare the results with the existing architectures. It is experimentally verified that the proposed model outperforms the state-of-the-art FIR GNN model by 9.2% and 14% in terms of SW and BW localization, respectively.
Learning to learn online with neuromodulated synaptic plasticity in spiking neural networks
Authors: Samuel Schmidgall, Joe Hays
Date: 2022-06-25 00:00:00
Tasks: BIG-bench Machine Learning

We propose that in order to harness our understanding of neuroscience toward machine learning, we must first have powerful tools for training brain-like models of learning. Although substantial progress has been made toward understanding the dynamics of learning in the brain, neuroscience-derived models of learning have yet to demonstrate the same performance capabilities as methods in deep learning such as gradient descent. Inspired by the successes of machine learning using gradient descent, we demonstrate that models of neuromodulated synaptic plasticity from neuroscience can be trained in Spiking Neural Networks (SNNs) with a framework of learning to learn through gradient descent to address challenging online learning problems. This framework opens a new path toward developing neuroscience inspired online learning algorithms.
Domain Generalization with Relaxed Instance Frequency-wise Normalization for Multi-device Acoustic Scene Classification
Authors: Byeonggeun Kim, Seunghan Yang, Jangho Kim, Hyunsin Park, JunTae Lee, Simyung Chang
Date: 2022-06-24 00:00:00
Tasks: Acoustic Scene Classification, Domain Generalization, Scene Classification

While using two-dimensional convolutional neural networks (2D-CNNs) in image processing, it is possible to manipulate domain information using channel statistics, and instance normalization has been a promising way to get domain-invariant features. However, unlike image processing, we analyze that domain-relevant information in an audio feature is dominant in frequency statistics rather than channel statistics. Motivated by our analysis, we introduce Relaxed Instance Frequency-wise Normalization (RFN): a plug-and-play, explicit normalization module along the frequency axis which can eliminate instance-specific domain discrepancy in an audio feature while relaxing undesirable loss of useful discriminative information. Empirically, simply adding RFN to networks shows clear margins compared to previous domain generalization approaches on acoustic scene classification and yields improved robustness for multiple audio devices. Especially, the proposed RFN won the DCASE2021 challenge TASK1A, low-complexity acoustic scene classification with multiple devices, with a clear margin, and RFN is an extended work of our technical report.
Black Box Optimization Using QUBO and the Cross Entropy Method
Authors: Jonas Nüßlein, Christoph Roch, Thomas Gabor, Jonas Stein, Claudia Linnhoff-Popien, Sebastian Feld
Date: 2022-06-24 00:00:00

Black-box optimization (BBO) can be used to optimize functions whose analytic form is unknown. A common approach to realising BBO is to learn a surrogate model which approximates the target black-box function which can then be solved via white-box optimization methods. In this paper, we present our approach BOX-QUBO, where the surrogate model is a QUBO matrix. However, unlike in previous state-of-the-art approaches, this matrix is not trained entirely by regression, but mostly by classification between 'good' and 'bad' solutions. This better accounts for the low capacity of the QUBO matrix, resulting in significantly better solutions overall. We tested our approach against the state-of-the-art on four domains and in all of them BOX-QUBO showed better results. A second contribution of this paper is the idea to also solve white-box problems, i.e. problems which could be directly formulated as QUBO, by means of black-box optimization in order to reduce the size of the QUBOs to the information-theoretic minimum. Experiments show that this significantly improves the results for MAX-k-SAT.
Multitask vocal burst modeling with ResNets and pre-trained paralinguistic Conformers
Authors: Josh Belanich, Krishna Somandepalli, Brian Eoff, Brendan Jou
Date: 2022-06-24 00:00:00
Tasks: Emotion Recognition, Event Detection, image-classification, Image Classification, Sound Event Detection, Speech Emotion Recognition

This technical report presents the modeling approaches used in our submission to the ICML Expressive Vocalizations Workshop & Competition multitask track (ExVo-MultiTask). We first applied image classification models of various sizes on mel-spectrogram representations of the vocal bursts, as is standard in sound event detection literature. Results from these models show an increase of 21.24% over the baseline system with respect to the harmonic mean of the task metrics, and comprise our team's main submission to the MultiTask track. We then sought to characterize the headroom in the MultiTask track by applying a large pre-trained Conformer model that previously achieved state-of-the-art results on paralinguistic tasks like speech emotion recognition and mask detection. We additionally investigated the relationship between the sub-tasks of emotional expression, country of origin, and age prediction, and discovered that the best performing models are trained as single-task models, questioning whether the problem truly benefits from a multitask setting.
Arithmetic Circuits, Structured Matrices and (not so) Deep Learning
Authors: Atri Rudra
Date: 2022-06-24 00:00:00
Tasks: Deep Learning, Survey

This survey presents a necessarily incomplete (and biased) overview of results at the intersection of arithmetic circuit complexity, structured matrices and deep learning. Recently there has been some research activity in replacing unstructured weight matrices in neural networks by structured ones (with the aim of reducing the size of the corresponding deep learning models). Most of this work has been experimental and in this survey, we formalize the research question and show how a recent work that combines arithmetic circuit complexity, structured matrices and deep learning essentially answers this question. This survey is targeted at complexity theorists who might enjoy reading about how tools developed in arithmetic circuit complexity helped design (to the best of our knowledge) a new family of structured matrices, which in turn seem well-suited for applications in deep learning. However, we hope that folks primarily interested in deep learning would also appreciate the connections to complexity theory.
Variational Bayesian inference for CP tensor completion with side information
Authors: Stanislav Budzinskiy, Nikolai Zamarashkin
Date: 2022-06-24 00:00:00
Tasks: Bayesian Inference

We propose a message passing algorithm, based on variational Bayesian inference, for low-rank tensor completion with automatic rank determination in the canonical polyadic format when additional side information (SI) is given. The SI comes in the form of low-dimensional subspaces the contain the fiber spans of the tensor (columns, rows, tubes, etc.). We validate the regularization properties induced by SI with extensive numerical experiments on synthetic and real-world data and present the results about tensor recovery and rank determination. The results show that the number of samples required for successful completion is significantly reduced in the presence of SI. We also discuss the origin of a bump in the phase transition curves that exists when the dimensionality of SI is comparable with that of the tensor.