pwc-archive/papers-with-abstracts huggingface dataset:
Transfer Learning for Control Systems via Neural Simulation Relations
Authors: Alireza Nadali, Bingzhuo Zhong, Ashutosh Trivedi, Majid Zamani
Date: 2024-12-02 00:00:00
Tasks: Transfer Learning

Transfer learning is an umbrella term for machine learning approaches that leverage knowledge gained from solving one problem (the source domain) to improve speed, efficiency, and data requirements in solving a different but related problem (the target domain). The performance of the transferred model in the target domain is typically measured via some notion of loss function in the target domain. This paper focuses on effectively transferring control logic from a source control system to a target control system while providing approximately similar behavioral guarantees in both domains. However, in the absence of a complete characterization of behavioral specifications, this problem cannot be captured in terms of loss functions. To overcome this challenge, we use (approximate) simulation relations to characterize observational equivalence between the behaviors of two systems. Simulation relations ensure that the outputs of both systems, equipped with their corresponding controllers, remain close to each other over time, and their closeness can be quantified {\it a priori}. By parameterizing simulation relations with neural networks, we introduce the notion of \emph{neural simulation relations}, which provides a data-driven approach to transfer any synthesized controller, regardless of the specification of interest, along with its proof of correctness. Compared with prior approaches, our method eliminates the need for a closed-loop mathematical model and specific requirements for both the source and target systems. We also introduce validity conditions that, when satisfied, guarantee the closeness of the outputs of two systems equipped with their corresponding controllers, thus eliminating the need for post-facto verification. We demonstrate the effectiveness of our approach through case studies involving a vehicle and a double inverted pendulum.
FERERO: A Flexible Framework for Preference-Guided Multi-Objective Learning
Authors: Lisha Chen, AFM Saif, Yanning Shen, Tianyi Chen
Date: 2024-12-02 00:00:00

Finding specific preference-guided Pareto solutions that represent different trade-offs among multiple objectives is critical yet challenging in multi-objective problems. Existing methods are restrictive in preference definitions and/or their theoretical guarantees. In this work, we introduce a Flexible framEwork for pREfeRence-guided multi-Objective learning (FERERO) by casting it as a constrained vector optimization problem. Specifically, two types of preferences are incorporated into this formulation -- the relative preference defined by the partial ordering induced by a polyhedral cone, and the absolute preference defined by constraints that are linear functions of the objectives. To solve this problem, convergent algorithms are developed with both single-loop and stochastic variants. Notably, this is the first single-loop primal algorithm for constrained vector optimization to our knowledge. The proposed algorithms adaptively adjust to both constraint and objective values, eliminating the need to solve different subproblems at different stages of constraint satisfaction. Experiments on multiple benchmarks demonstrate the proposed method is very competitive in finding preference-guided optimal solutions. Code is available at https://github.com/lisha-chen/FERERO/.
Bluetooth Low Energy Dataset Using In-Phase and Quadrature Samples for Indoor Localization
Authors: Samuel G. Leitch, Qasim Zeeshan Ahmed, Ben Van Herbruggen, Mathias Baert, Jaron Fontaine, Eli de Poorter, Adnan Shahid, Pavlos I. Lazaridis
Date: 2024-12-02 00:00:00
Tasks: Indoor Localization

One significant challenge in research is to collect a large amount of data and learn the underlying relationship between the input and the output variables. This paper outlines the process of collecting and validating a dataset designed to determine the angle of arrival (AoA) using Bluetooth low energy (BLE) technology. The data, collected in a laboratory setting, is intended to approximate real-world industrial scenarios. This paper discusses the data collection process, the structure of the dataset, and the methodology adopted for automating sample labeling for supervised learning. The collected samples and the process of generating ground truth (GT) labels were validated using the Texas Instruments (TI) phase difference of arrival (PDoA) implementation on the data, yielding a mean absolute error (MAE) at one of the heights without obstacles of $25.71^\circ$. The distance estimation on BLE was implemented using a Gaussian Process Regression algorithm, yielding an MAE of $0.174$m.
The Data-Driven Censored Newsvendor Problem
Authors: Chamsi Hssaine, Sean R. Sinclair
Date: 2024-12-02 00:00:00

We study a censored variant of the data-driven newsvendor problem, where the decision-maker must select an ordering quantity that minimizes expected overage and underage costs based only on offline censored sales data, rather than historical demand realizations. Our goal is to understand how the degree of historical demand censoring affects the performance of any learning algorithm for this problem. To isolate this impact, we adopt a distributionally robust optimization framework, evaluating policies according to their worst-case regret over an ambiguity set of distributions. This set is defined by the largest historical order quantity (the observable boundary of the dataset), and contains all distributions matching the true demand distribution up to this boundary, while allowing them to be arbitrary afterwards. We demonstrate a spectrum of achievability under demand censoring by deriving a natural necessary and sufficient condition under which vanishing regret is an achievable goal. In regimes in which it is not, we exactly characterize the information loss due to censoring: an insurmountable lower bound on the performance of any policy, even when the decision-maker has access to infinitely many demand samples. We then leverage these sharp characterizations to propose a natural robust algorithm that adapts to the historical level of demand censoring. We derive finite-sample guarantees for this algorithm across all possible censoring regimes and show its near-optimality with matching lower bounds (up to polylogarithmic factors). We moreover demonstrate its robust performance via extensive numerical experiments on both synthetic and real-world datasets.
Structure-Guided Input Graph for GNNs facing Heterophily
Authors: Victor M. Tenorio, Madeline Navarro, Samuel Rey, Santiago Segarra, Antonio G. Marques
Date: 2024-12-02 00:00:00

Graph Neural Networks (GNNs) have emerged as a promising tool to handle data exhibiting an irregular structure. However, most GNN architectures perform well on homophilic datasets, where the labels of neighboring nodes are likely to be the same. In recent years, an increasing body of work has been devoted to the development of GNN architectures for heterophilic datasets, where labels do not exhibit this low-pass behavior. In this work, we create a new graph in which nodes are connected if they share structural characteristics, meaning a higher chance of sharing their labels, and then use this new graph in the GNN architecture. To do this, we compute the k-nearest neighbors graph according to distances between structural features, which are either (i) role-based, such as degree, or (ii) global, such as centrality measures. Experiments show that the labels are smoother in this newly defined graph and that the performance of GNN architectures improves when using this alternative structure.
Adversarial Sample-Based Approach for Tighter Privacy Auditing in Final Model-Only Scenarios
Authors: Sangyeon Yoon, Wonje Jeung, Albert No
Date: 2024-12-02 00:00:00

Auditing Differentially Private Stochastic Gradient Descent (DP-SGD) in the final model setting is challenging and often results in empirical lower bounds that are significantly looser than theoretical privacy guarantees. We introduce a novel auditing method that achieves tighter empirical lower bounds without additional assumptions by crafting worst-case adversarial samples through loss-based input-space auditing. Our approach surpasses traditional canary-based heuristics and is effective in final model-only scenarios. Specifically, with a theoretical privacy budget of $\varepsilon = 10.0$, our method achieves empirical lower bounds of $4.914$, compared to the baseline of $4.385$ for MNIST. Our work offers a practical framework for reliable and accurate privacy auditing in differentially private machine learning.
CBOL-Tuner: Classifier-pruned Bayesian optimization to explore temporally structured latent spaces for particle accelerator tuning
Authors: Mahindra Rautela, Alan Williams, Alexander Scheinker
Date: 2024-12-02 00:00:00
Tasks: Bayesian Optimization, Efficient Exploration, global-optimization, parameter estimation

Complex dynamical systems, such as particle accelerators, often require complicated and time-consuming tuning procedures for optimal performance. It may also be required that these procedures estimate the optimal system parameters, which govern the dynamics of a spatiotemporal beam -- this can be a high-dimensional optimization problem. To address this, we propose a Classifier-pruned Bayesian Optimization-based Latent space Tuner (CBOL-Tuner), a framework for efficient exploration within a temporally-structured latent space. The CBOL-Tuner integrates a convolutional variational autoencoder (CVAE) for latent space representation, a long short-term memory (LSTM) network for temporal dynamics, a dense neural network (DNN) for parameter estimation, and a classifier-pruned Bayesian optimizer (C-BO) to adaptively search and filter the latent space for optimal solutions. CBOL-Tuner demonstrates superior performance in identifying multiple optimal settings and outperforms alternative global optimization methods.
Query Performance Explanation through Large Language Model for HTAP Systems
Authors: Haibo Xiu, Li Zhang, Tieying Zhang, Jun Yang, Jianjun Chen
Date: 2024-12-02 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model, RAG, Retrieval, Retrieval-augmented Generation

In hybrid transactional and analytical processing (HTAP) systems, users often struggle to understand why query plans from one engine (OLAP or OLTP) perform significantly slower than those from another. Although optimizers provide plan details via the EXPLAIN function, these explanations are frequently too technical for non-experts and offer limited insights into performance differences across engines. To address this, we propose a novel framework that leverages large language models (LLMs) to explain query performance in HTAP systems. Built on Retrieval-Augmented Generation (RAG), our framework constructs a knowledge base that stores historical query executions and expert-curated explanations. To enable efficient retrieval of relevant knowledge, query plans are embedded using a lightweight tree-CNN classifier. This augmentation allows the LLM to generate clear, context-aware explanations of performance differences between engines. Our approach demonstrates the potential of LLMs in hybrid engine systems, paving the way for further advancements in database optimization and user support.
Review of Mathematical Optimization in Federated Learning
Authors: Shusen Yang, Fangyuan Zhao, ZiHao Zhou, Liang Shi, Xuebin Ren, Zongben Xu
Date: 2024-12-02 00:00:00
Tasks: Distributed Optimization, Federated Learning

Federated Learning (FL) has been becoming a popular interdisciplinary research area in both applied mathematics and information sciences. Mathematically, FL aims to collaboratively optimize aggregate objective functions over distributed datasets while satisfying a variety of privacy and system constraints.Different from conventional distributed optimization methods, FL needs to address several specific issues (e.g., non-i.i.d. data distributions and differential private noises), which pose a set of new challenges in the problem formulation, algorithm design, and convergence analysis. In this paper, we will systematically review existing FL optimization research including their assumptions, formulations, methods, and theoretical results. Potential future directions are also discussed.
Representation and Regression Problems in Neural Networks: Relaxation, Generalization, and Numerics
Authors: Kang Liu, Enrique Zuazua
Date: 2024-12-02 00:00:00
Tasks: Generalization Bounds, regression

In this work, we address three non-convex optimization problems associated with the training of shallow neural networks (NNs) for exact and approximate representation, as well as for regression tasks. Through a mean-field approach, we convexify these problems and, applying a representer theorem, prove the absence of relaxation gaps. We establish generalization bounds for the resulting NN solutions, assessing their predictive performance on test datasets and, analyzing the impact of key hyperparameters on these bounds, propose optimal choices. On the computational side, we examine the discretization of the convexified problems and derive convergence rates. For low-dimensional datasets, these discretized problems are efficiently solvable using the simplex method. For high-dimensional datasets, we propose a sparsification algorithm that, combined with gradient descent for over-parameterized shallow NNs, yields effective solutions to the primal problems.
Characterizing Jupiter's interior using machine learning reveals four key structures
Authors: Maayan Ziv, Eli Galanti, Saburo Howard, Tristan Guillot, Yohai Kaspi
Date: 2024-12-02 00:00:00

The internal structure of Jupiter is constrained by the precise gravity field measurements by NASA's Juno mission, atmospheric data from the Galileo entry probe, and Voyager radio occultations. Not only are these observations few compared to the possible interior setups and their multiple controlling parameters, but they remain challenging to reconcile. As a complex, multidimensional problem, characterizing typical structures can help simplify the modeling process. We used NeuralCMS, a deep learning model based on the accurate concentric Maclaurin spheroid (CMS) method, coupled with a fully consistent wind model to efficiently explore a wide range of interior models without prior assumptions. We then identified those consistent with the measurements and clustered the plausible combinations of parameters controlling the interior. We determine the plausible ranges of internal structures and the dynamical contributions to Jupiter's gravity field. Four typical interior structures are identified, characterized by their envelope and core properties. This reduces the dimensionality of Jupiter's interior to only two effective parameters. Within the reduced 2D phase space, we show that the most observationally constrained structures fall within one of the key structures, but they require a higher 1 bar temperature than the observed value. We provide a robust framework for characterizing giant planet interiors with consistent wind treatment, demonstrating that for Jupiter, wind constraints strongly impact the gravity harmonics while the interior parameter distribution remains largely unchanged. Importantly, we find that Jupiter's interior can be described by two effective parameters that clearly distinguish the four characteristic structures and conclude that atmospheric measurements may not fully represent the entire envelope.
Kernel-Based Optimal Control: An Infinitesimal Generator Approach
Authors: Petar Bevanda, Nicolas Hoischen, Tobias Wittmann, Jan Brüdigam, Sandra Hirche, Boris Houska
Date: 2024-12-02 00:00:00

This paper presents a novel operator-theoretic approach for optimal control of nonlinear stochastic systems within reproducing kernel Hilbert spaces. Our learning framework leverages data samples of system dynamics and stage cost functions, with only control penalties and constraints provided. The proposed method directly learns the infinitesimal generator of a controlled stochastic diffusion in an infinite-dimensional hypothesis space. We demonstrate that our approach seamlessly integrates with modern convex operator-theoretic Hamilton-Jacobi-Bellman recursions, enabling a data-driven solution to the optimal control problems. Furthermore, our learning framework includes nonparametric estimators for uncontrolled infinitesimal generators as a special case. Numerical experiments, ranging from synthetic differential equations to simulated robotic systems, showcase the advantages of our approach compared to both modern data-driven and classical nonlinear programming methods for optimal control.
Unifying AMP Algorithms for Rotationally-Invariant Models
Authors: Songbin Liu, Junjie Ma
Date: 2024-12-02 00:00:00

This paper presents a unified framework for constructing Approximate Message Passing (AMP) algorithms for rotationally-invariant models. By employing a general iterative algorithm template and reducing it to long-memory Orthogonal AMP (OAMP), we systematically derive the correct Onsager terms of AMP algorithms. This approach allows us to rederive an AMP algorithm introduced by Fan and Opper et al., while shedding new light on the role of free cumulants of the spectral law. The free cumulants arise naturally from a recursive centering operation, potentially of independent interest beyond the scope of AMP. To illustrate the flexibility of our framework, we introduce two novel AMP variants and apply them to estimation in spiked models.
Tokenizing 3D Molecule Structure with Quantized Spherical Coordinates
Authors: Kaiyuan Gao, Yusong Wang, Haoxiang Guan, Zun Wang, Qizhi Pei, John E. Hopcroft, Kun He, Lijun Wu
Date: 2024-12-02 00:00:00
Tasks: molecular representation, Property Prediction

The application of language models (LMs) to molecular structure generation using line notations such as SMILES and SELFIES has been well-established in the field of cheminformatics. However, extending these models to generate 3D molecular structures presents significant challenges. Two primary obstacles emerge: (1) the difficulty in designing a 3D line notation that ensures SE(3)-invariant atomic coordinates, and (2) the non-trivial task of tokenizing continuous coordinates for use in LMs, which inherently require discrete inputs. To address these challenges, we propose Mol-StrucTok, a novel method for tokenizing 3D molecular structures. Our approach comprises two key innovations: (1) We design a line notation for 3D molecules by extracting local atomic coordinates in a spherical coordinate system. This notation builds upon existing 2D line notations and remains agnostic to their specific forms, ensuring compatibility with various molecular representation schemes. (2) We employ a Vector Quantized Variational Autoencoder (VQ-VAE) to tokenize these coordinates, treating them as generation descriptors. To further enhance the representation, we incorporate neighborhood bond lengths and bond angles as understanding descriptors. Leveraging this tokenization framework, we train a GPT-2 style model for 3D molecular generation tasks. Results demonstrate strong performance with significantly faster generation speeds and competitive chemical stability compared to previous methods. Further, by integrating our learned discrete representations into Graphormer model for property prediction on QM9 dataset, Mol-StrucTok reveals consistent improvements across various molecular properties, underscoring the versatility and robustness of our approach.
FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
Authors: Yujie Wang, Shiju Wang, Shenhan Zhu, Fangcheng Fu, Xinyi Liu, Xuefeng Xiao, Huixia Li, Jiashi Li, Faming Wu, Bin Cui
Date: 2024-12-02 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model

Extending the context length (i.e., the maximum supported sequence length) of LLMs is of paramount significance. To facilitate long context training of LLMs, sequence parallelism has emerged as an essential technique, which scatters each input sequence across multiple devices and necessitates communication to process the sequence. In essence, existing sequence parallelism methods assume homogeneous sequence lengths (i.e., all input sequences are equal in length) and therefore leverages a single, static scattering strategy for all input sequences. However, in reality, the sequence lengths in LLM training corpora exhibit substantial variability, often following a long-tail distribution, which leads to workload heterogeneity. In this paper, we show that employing a single, static strategy results in inefficiency and resource under-utilization, highlighting the need for adaptive approaches to handle the heterogeneous workloads across sequences. To address this, we propose a heterogeneity-adaptive sequence parallelism method. For each training step, our approach captures the variability in sequence lengths and assigns the optimal combination of scattering strategies based on workload characteristics. We model this problem as a linear programming optimization and design an efficient and effective solver to find the optimal solution. Furthermore, we implement our method in a high-performance system that supports adaptive parallelization in distributed LLM training. Experimental results demonstrate that our system outperforms state-of-the-art training frameworks by up to 1.98x.
ReHub: Linear Complexity Graph Transformers with Adaptive Hub-Spoke Reassignment
Authors: Tomer Borreda, Daniel Freedman, Or Litany
Date: 2024-12-02 00:00:00
Tasks: Graph Learning

We present ReHub, a novel graph transformer architecture that achieves linear complexity through an efficient reassignment technique between nodes and virtual nodes. Graph transformers have become increasingly important in graph learning for their ability to utilize long-range node communication explicitly, addressing limitations such as oversmoothing and oversquashing found in message-passing graph networks. However, their dense attention mechanism scales quadratically with the number of nodes, limiting their applicability to large-scale graphs. ReHub draws inspiration from the airline industry's hub-and-spoke model, where flights are assigned to optimize operational efficiency. In our approach, graph nodes (spokes) are dynamically reassigned to a fixed number of virtual nodes (hubs) at each model layer. Recent work, Neural Atoms (Li et al., 2024), has demonstrated impressive and consistent improvements over GNN baselines by utilizing such virtual nodes; their findings suggest that the number of hubs strongly influences performance. However, increasing the number of hubs typically raises complexity, requiring a trade-off to maintain linear complexity. Our key insight is that each node only needs to interact with a small subset of hubs to achieve linear complexity, even when the total number of hubs is large. To leverage all hubs without incurring additional computational costs, we propose a simple yet effective adaptive reassignment technique based on hub-hub similarity scores, eliminating the need for expensive node-hub computations. Our experiments on LRGB indicate a consistent improvement in results over the base method, Neural Atoms, while maintaining a linear complexity. Remarkably, our sparse model achieves performance on par with its non-sparse counterpart. Furthermore, ReHub outperforms competitive baselines and consistently ranks among top performers across various benchmarks.