pwc-archive/papers-with-abstracts huggingface dataset:
Anticipatory Music Transformer
Authors: John Thickstun, David Hall, Chris Donahue, Percy Liang
Date: 2023-06-14 00:00:00
Tasks: Music Generation

We introduce anticipation: a method for constructing a controllable generative model of a temporal point process (the event process) conditioned asynchronously on realizations of a second, correlated process (the control process). We achieve this by interleaving sequences of events and controls, such that controls appear following stopping times in the event sequence. This work is motivated by problems arising in the control of symbolic music generation. We focus on infilling control tasks, whereby the controls are a subset of the events themselves, and conditional generation completes a sequence of events given the fixed control events. We train anticipatory infilling models using the large and diverse Lakh MIDI music dataset. These models match the performance of autoregressive models for prompted music generation, with the additional capability to perform infilling control tasks, including accompaniment. Human evaluators report that an anticipatory model produces accompaniments with similar musicality to even music composed by humans over a 20-second clip.
Multi-class Graph Clustering via Approximated Effective $p$-Resistance
Authors: Shota Saito, Mark Herbster
Date: 2023-06-14 00:00:00
Tasks: Clustering, Graph Clustering

This paper develops an approximation to the (effective) $p$-resistance and applies it to multi-class clustering. Spectral methods based on the graph Laplacian and its generalization to the graph $p$-Laplacian have been a backbone of non-euclidean clustering techniques. The advantage of the $p$-Laplacian is that the parameter $p$ induces a controllable bias on cluster structure. The drawback of $p$-Laplacian eigenvector based methods is that the third and higher eigenvectors are difficult to compute. Thus, instead, we are motivated to use the $p$-resistance induced by the $p$-Laplacian for clustering. For $p$-resistance, small $p$ biases towards clusters with high internal connectivity while large $p$ biases towards clusters of small "extent," that is a preference for smaller shortest-path distances between vertices in the cluster. However, the $p$-resistance is expensive to compute. We overcome this by developing an approximation to the $p$-resistance. We prove upper and lower bounds on this approximation and observe that it is exact when the graph is a tree. We also provide theoretical justification for the use of $p$-resistance for clustering. Finally, we provide experiments comparing our approximated $p$-resistance clustering to other $p$-Laplacian based methods.
TensorKrowch: Smooth integration of tensor networks in machine learning
Authors: José Ramón Pareja Monturiol, David Pérez-García, Alejandro Pozas-Kerstjens
Date: 2023-06-14 00:00:00
Tasks: Tensor Networks

Tensor networks are factorizations of high-dimensional tensors into networks of smaller tensors. They have applications in physics and mathematics, and recently have been proposed as promising machine learning architectures. To ease the integration of tensor networks in machine learning pipelines, we introduce TensorKrowch, an open source Python library built on top of PyTorch. Providing a user-friendly interface, TensorKrowch allows users to construct any tensor network, train it, and integrate it as a layer in more intricate deep learning models. In this paper, we describe the main functionality and basic usage of TensorKrowch, and provide technical details on its building blocks and the optimizations performed to achieve efficient operation.
Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning
Authors: Nikhil Vyas, Depen Morwani, Rosie Zhao, Gal Kaplun, Sham Kakade, Boaz Barak
Date: 2023-06-14 00:00:00

The success of SGD in deep learning has been ascribed by prior works to the implicit bias induced by finite batch sizes ("SGD noise"). While prior works focused on offline learning (i.e., multiple-epoch training), we study the impact of SGD noise on online (i.e., single epoch) learning. Through an extensive empirical analysis of image and language data, we demonstrate that small batch sizes do not confer any implicit bias advantages in online learning. In contrast to offline learning, the benefits of SGD noise in online learning are strictly computational, facilitating more cost-effective gradient steps. This suggests that SGD in the online regime can be construed as taking noisy steps along the "golden path" of the noiseless gradient descent algorithm. We study this hypothesis and provide supporting evidence in loss and function space. Our findings challenge the prevailing understanding of SGD and offer novel insights into its role in online learning.
Federated Learning-based Vehicle Trajectory Prediction against Cyberattacks
Authors: Zhe Wang, Tingkai Yan
Date: 2023-06-14 00:00:00
Tasks: Federated Learning, Prediction, Trajectory Prediction

With the development of the Internet of Vehicles (IoV), vehicle wireless communication poses serious cybersecurity challenges. Faulty information, such as fake vehicle positions and speeds sent by surrounding vehicles, could cause vehicle collisions, traffic jams, and even casualties. Additionally, private vehicle data leakages, such as vehicle trajectory and user account information, may damage user property and security. Therefore, achieving a cyberattack-defense scheme in the IoV system with faulty data saturation is necessary. This paper proposes a Federated Learning-based Vehicle Trajectory Prediction Algorithm against Cyberattacks (FL-TP) to address the above problems. The FL-TP is intensively trained and tested using a publicly available Vehicular Reference Misbehavior (VeReMi) dataset with five types of cyberattacks: constant, constant offset, random, random offset, and eventual stop. The results show that the proposed FL-TP algorithm can improve cyberattack detection and trajectory prediction by up to 6.99% and 54.86%, respectively, under the maximum cyberattack permeability scenarios compared with benchmark methods.
Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approach
Authors: Hongyang R. Zhang, Dongyue Li, Haotian Ju
Date: 2023-06-14 00:00:00
Tasks: Data Augmentation, image-classification, Image Classification, Stochastic Optimization

The training of over-parameterized neural networks has received much study in recent literature. An important consideration is the regularization of over-parameterized networks due to their highly nonconvex and nonlinear geometry. In this paper, we study noise injection algorithms, which can regularize the Hessian of the loss, leading to regions with flat loss surfaces. Specifically, by injecting isotropic Gaussian noise into the weight matrices of a neural network, we can obtain an approximately unbiased estimate of the trace of the Hessian. However, naively implementing the noise injection via adding noise to the weight matrices before backpropagation presents limited empirical improvements. To address this limitation, we design a two-point estimate of the Hessian penalty, which injects noise into the weight matrices along both positive and negative directions of the random noise. In particular, this two-point estimate eliminates the variance of the first-order Taylor's expansion term on the Hessian. We show a PAC-Bayes generalization bound that depends on the trace of the Hessian (and the radius of the weight space), which can be measured from data. We conduct a detailed experimental study to validate our approach and show that it can effectively regularize the Hessian and improve generalization. First, our algorithm can outperform prior approaches on sharpness-reduced training, delivering up to a 2.4% test accuracy increase for fine-tuning ResNets on six image classification datasets. Moreover, the trace of the Hessian reduces by 15.8%, and the largest eigenvalue is reduced by 9.7% with our approach. We also find that the regularization of the Hessian can be combined with weight decay and data augmentation, leading to stronger regularization. Second, our approach remains effective for improving generalization in pretraining multimodal CLIP models and chain-of-thought fine-tuning.
Fast and Private Inference of Deep Neural Networks by Co-designing Activation Functions
Authors: Abdulrahman Diaa, Lucas Fenaux, Thomas Humphries, Marian Dietz, Faezeh Ebrahimianghazani, Bailey Kacsmar, Xinda Li, Nils Lukas, Rasoul Akhavan Mahdavi, Simon Oya, Ehsan Amjadian, Florian Kerschbaum
Date: 2023-06-14 00:00:00
Tasks: image-classification, Image Classification

Machine Learning as a Service (MLaaS) is an increasingly popular design where a company with abundant computing resources trains a deep neural network and offers query access for tasks like image classification. The challenge with this design is that MLaaS requires the client to reveal their potentially sensitive queries to the company hosting the model. Multi-party computation (MPC) protects the client's data by allowing encrypted inferences. However, current approaches suffer from prohibitively large inference times. The inference time bottleneck in MPC is the evaluation of non-linear layers such as ReLU activation functions. Motivated by the success of previous work co-designing machine learning and MPC, we develop an activation function co-design. We replace all ReLUs with a polynomial approximation and evaluate them with single-round MPC protocols, which give state-of-the-art inference times in wide-area networks. Furthermore, to address the accuracy issues previously encountered with polynomial activations, we propose a novel training algorithm that gives accuracy competitive with plaintext models. Our evaluation shows between $3$ and $110\times$ speedups in inference time on large models with up to $23$ million parameters while maintaining competitive inference accuracy.
SQL2Circuits: Estimating Metrics for SQL Queries with a Quantum Natural Language Processing Method
Authors: Valter Uotila
Date: 2023-06-14 00:00:00
Tasks: Binary Classification, Quantum Machine Learning

In recent years, advances in quantum computing have led to accelerating research on quantum applications across fields. Here, we introduce a quantum machine learning model as a potential solution to the classical question in database research: the estimation of metrics for SQL queries. This work employs a quantum natural language processing (QNLP)-inspired approach for constructing a quantum machine learning model that can classify SQL queries with respect to their cardinalities, costs, and execution times. The model consists of an encoding mechanism and a training phase, including classical and quantum subroutines. The encoding mechanism encodes SQL queries as parametrized quantum circuits. In the training phase, we utilize classical optimization algorithms, such as SPSA and Adam, to optimize the circuit parameters to make predictions about the query metrics. We conclude that our model reaches an accuracy equivalent to that of the QNLP model in the binary classification tasks. Moreover, we extend the previous work by adding 4-class classification tasks and compare the cardinality estimation results to the state-of-the-art databases. We perform a theoretical analysis of the quantum machine learning model by calculating its expressibility and entangling capabilities. The analysis shows that the model has advantageous properties that make it expressible but also not too complex to be executed on the existing quantum hardware.
Permutation Invariant Recurrent Neural Networks for Sound Source Tracking Applications
Authors: David Diaz-Guerra, Archontis Politis, Antonio Miguel, Jose R. Beltran, Tuomas Virtanen
Date: 2023-06-14 00:00:00

Many multi-source localization and tracking models based on neural networks use one or several recurrent layers at their final stages to track the movement of the sources. Conventional recurrent neural networks (RNNs), such as the long short-term memories (LSTMs) or the gated recurrent units (GRUs), take a vector as their input and use another vector to store their state. However, this approach results in the information from all the sources being contained in a single ordered vector, which is not optimal for permutation-invariant problems such as multi-source tracking. In this paper, we present a new recurrent architecture that uses unordered sets to represent both its input and its state and that is invariant to the permutations of the input set and equivariant to the permutations of the state set. Hence, the information of every sound source is represented in an individual embedding and the new estimates are assigned to the tracked trajectories regardless of their order.
DiffuDetox: A Mixed Diffusion Model for Text Detoxification
Authors: Griffin Floto, Mohammad Mahdi Abdollah Pour, Parsa Farinneya, Zhenwei Tang, Ali Pesaranghader, Manasa Bharadwaj, Scott Sanner
Date: 2023-06-14 00:00:00
Tasks: Conditional Text Generation, Diversity, model, Text Generation

Text detoxification is a conditional text generation task aiming to remove offensive content from toxic text. It is highly useful for online forums and social media, where offensive content is frequently encountered. Intuitively, there are diverse ways to detoxify sentences while preserving their meanings, and we can select from detoxified sentences before displaying text to users. Conditional diffusion models are particularly suitable for this task given their demonstrated higher generative diversity than existing conditional text generation models based on language models. Nonetheless, text fluency declines when they are trained with insufficient data, which is the case for this task. In this work, we propose DiffuDetox, a mixed conditional and unconditional diffusion model for text detoxification. The conditional model takes toxic text as the condition and reduces its toxicity, yielding a diverse set of detoxified sentences. The unconditional model is trained to recover the input text, which allows the introduction of additional fluent text for training and thus ensures text fluency. Extensive experimental results and in-depth analysis demonstrate the effectiveness of our proposed DiffuDetox.
Adaptive Modeling of Satellite-Derived Nighttime Lights Time-Series for Tracking Urban Change Processes Using Machine Learning
Authors: Srija Chakraborty, Eleanor C. Stokes
Date: 2023-06-14 00:00:00
Tasks: Anomaly Detection, Decision Making, Time Series

Remotely sensed nighttime lights (NTL) uniquely capture urban change processes that are important to human and ecological well-being, such as urbanization, socio-political conflicts and displacement, impacts from disasters, holidays, and changes in daily human patterns of movement. Though several NTL products are global in extent, intrinsic city-specific factors that affect lighting, such as development levels, and social, economic, and cultural characteristics, are unique to each city, making the urban processes embedded in NTL signatures difficult to characterize, and limiting the scalability of urban change analyses. In this study, we propose a data-driven approach to detect urban changes from daily satellite-derived NTL data records that is adaptive across cities and effective at learning city-specific temporal patterns. The proposed method learns to forecast NTL signatures from past data records using neural networks and allows the use of large volumes of unlabeled data, eliminating annotation effort. Urban changes are detected based on deviations of observed NTL from model forecasts using an anomaly detection approach. Comparing model forecasts with observed NTL also allows identifying the direction of change (positive or negative) and monitoring change severity for tracking recovery. In operationalizing the model, we consider ten urban areas from diverse geographic regions with dynamic NTL time-series and demonstrate the generalizability of the approach for detecting the change processes with different drivers and rates occurring within these urban areas based on NTL deviation. This scalable approach for monitoring changes from daily remote sensing observations efficiently utilizes large data volumes to support continuous monitoring and decision making.
Langevin Monte Carlo for strongly log-concave distributions: Randomized midpoint revisited
Authors: Lu Yu, Avetik Karagulyan, Arnak Dalalyan
Date: 2023-06-14 00:00:00

We revisit the problem of sampling from a target distribution that has a smooth strongly log-concave density everywhere in $\mathbb R^p$. In this context, if no additional density information is available, the randomized midpoint discretization for the kinetic Langevin diffusion is known to be the most scalable method in high dimensions with large condition numbers. Our main result is a nonasymptotic and easy to compute upper bound on the Wasserstein-2 error of this method. To provide a more thorough explanation of our method for establishing the computable upper bound, we conduct an analysis of the midpoint discretization for the vanilla Langevin process. This analysis helps to clarify the underlying principles and provides valuable insights that we use to establish an improved upper bound for the kinetic Langevin process with the midpoint discretization. Furthermore, by applying these techniques we establish new guarantees for the kinetic Langevin process with Euler discretization, which have a better dependence on the condition number than existing upper bounds.
Analysis and Approximate Inference of Large Random Kronecker Graphs
Authors: Zhenyu Liao, Yuanqian Xia, Chengmei Niu, Yong Xiao
Date: 2023-06-14 00:00:00
Tasks: Graph Classification, Recommendation Systems

Random graph models are playing an increasingly important role in various fields ranging from social networks, telecommunication systems, to physiologic and biological networks. Within this landscape, the random Kronecker graph model, emerges as a prominent framework for scrutinizing intricate real-world networks. In this paper, we investigate large random Kronecker graphs, i.e., the number of graph vertices $N$ is large. Built upon recent advances in random matrix theory (RMT) and high-dimensional statistics, we prove that the adjacency of a large random Kronecker graph can be decomposed, in a spectral norm sense, into two parts: a small-rank (of rank $O(\log N)$) signal matrix that is linear in the graph parameters and a zero-mean random noise matrix. Based on this result, we propose a ``denoise-and-solve'' approach to infer the key graph parameters, with significantly reduced computational complexity. Experiments on both graph inference and classification are presented to evaluate the our proposed method. In both tasks, the proposed approach yields comparable or advantageous performance, than widely-used graph inference (e.g., KronFit) and graph neural net baselines, at a time cost that scales linearly as the graph size $N$.
Bandits with Replenishable Knapsacks: the Best of both Worlds
Authors: Martino Bernasconi, Matteo Castiglioni, Andrea Celli, Federico Fusco
Date: 2023-06-14 00:00:00
Tasks: Decision Making

The bandits with knapsack (BwK) framework models online decision-making problems in which an agent makes a sequence of decisions subject to resource consumption constraints. The traditional model assumes that each action consumes a non-negative amount of resources and the process ends when the initial budgets are fully depleted. We study a natural generalization of the BwK framework which allows non-monotonic resource utilization, i.e., resources can be replenished by a positive amount. We propose a best-of-both-worlds primal-dual template that can handle any online learning problem with replenishment for which a suitable primal regret minimizer exists. In particular, we provide the first positive results for the case of adversarial inputs by showing that our framework guarantees a constant competitive ratio $\alpha$ when $B=\Omega(T)$ or when the possible per-round replenishment is a positive constant. Moreover, under a stochastic input model, our algorithm yields an instance-independent $\tilde{O}(T^{1/2})$ regret bound which complements existing instance-dependent bounds for the same setting. Finally, we provide applications of our framework to some economic problems of practical relevance.
Self-supervised Learning and Graph Classification under Heterophily
Authors: Yilin Ding, Zhen Liu, Hao Hao
Date: 2023-06-14 00:00:00
Tasks: Classification, Graph Classification, Graph Representation Learning, Molecular Property Prediction, Property Prediction, Protein Function Prediction, Representation Learning, Self-Supervised Learning

Self-supervised learning has shown its promising capability in graph representation learning in recent work. Most existing pre-training strategies usually choose the popular Graph neural networks (GNNs), which can be seen as a special form of low-pass filter, fail to effectively capture heterophily. In this paper, we first present an experimental investigation exploring the performance of low-pass and high-pass filters in heterophily graph classification, where the results clearly show that high-frequency signal is important for learning heterophily graph representation. On the other hand, it is still unclear how to effectively capture the structural pattern of graphs and how to measure the capability of the self-supervised pre-training strategy in capturing graph structure. To address the problem, we first design a quantitative metric to Measure Graph Structure (MGS), which analyzes correlation between structural similarity and embedding similarity of graph pairs. Then, to enhance the graph structural information captured by self-supervised learning, we propose a novel self-supervised strategy for Pre-training GNNs based on the Metric (PGM). Extensive experiments validate our pre-training strategy achieves state-of-the-art performance for molecular property prediction and protein function prediction. In addition, we find choosing the suitable filter sometimes may be better than designing good pre-training strategies for heterophily graph classification.
A Survey on Blood Pressure Measurement Technologies: Addressing Potential Sources of Bias
Authors: Seyedeh Somayyeh Mousavi, Matthew A. Reyna, Gari D. Clifford, Reza Sameni
Date: 2023-06-14 00:00:00
Tasks: Bayesian Inference, Management

Regular blood pressure (BP) monitoring in clinical and ambulatory settings plays a crucial role in the prevention, diagnosis, treatment, and management of cardiovascular diseases. Recently, the widespread adoption of ambulatory BP measurement devices has been driven predominantly by the increased prevalence of hypertension and its associated risks and clinical conditions. Recent guidelines advocate for regular BP monitoring as part of regular clinical visits or even at home. This increased utilization of BP measurement technologies has brought up significant concerns, regarding the accuracy of reported BP values across settings. In this survey, focusing mainly on cuff-based BP monitoring technologies, we highlight how BP measurements can demonstrate substantial biases and variances due to factors such as measurement and device errors, demographics, and body habitus. With these inherent biases, the development of a new generation of cuff-based BP devices which use artificial-intelligence (AI) has significant potential. We present future avenues where AI-assisted technologies can leverage the extensive clinical literature on BP-related studies together with the large collections of BP records available in electronic health records. These resources can be combined with machine learning approaches, including deep learning and Bayesian inference, to remove BP measurement biases and to provide individualized BP-related cardiovascular risk indexes.