pwc-archive/papers-with-abstracts huggingface dataset:
Tensor-based Model Reduction and Identification for Generalized Memory Polynomial
Authors: Yuchao Wang, Yimin Wei
Date: 2025-02-28 00:00:00
Tasks: Tensor Networks

Power amplifiers (PAs) are essential components in wireless communication systems, and the design of their behavioral models has been an important research topic for many years. The widely used generalized memory polynomial (GMP) model suffers from rapid growth in the number of parameters with increasing memory depths and nonlinearity order, which leads to a significant increase in model complexity and the risk of overfitting. In this study, we introduce tensor networks to compress the unknown coefficient tensor of the GMP model, resulting in three novel tensor-based GMP models. These models can achieve comparable performance to the GMP model, but with far fewer parameters and lower complexity. For the identification of these models, we derive the alternating least-squares (ALS) method to ensure the rapid updates and convergence of model parameters in an iterative manner. In addition, we notice that the horizontal slices of the third-order data tensor constructed from the input signals are Vandermonde matrices, which have a numerically low-rank structure. Hence, we further propose the RP-ALS algorithm, which first performs a truncated higher-order singular value decomposition on the data tensor to generate random projections, then conducts the ALS algorithm for the identification of projected models with downscaled dimensions, thus reducing the computational effort of the iterative process. The experimental results show that the proposed models outperform the full GMP model and sparse GMP model via LASSO regression in terms of the reduction in the number of parameters and running complexity.
A Study on the Specialist Predator with the Allee Effect on the Prey
Authors: Tanmay Das, Mahatsab Mandal
Date: 2025-03-03 00:00:00

Predator-prey models in theoretical ecology have a long and complex history, spanning decades of research. Most of the models rely upon simple reproduction and mortality rates associated with different types of functional responses. A key development in this field occurred with the introduction of a density dependent reproduction rate, originally introduced by Allee. In this manuscript, a new function representing the Allee effect is introduced and justified from the ecological point of view. This paper aims to analyze predator-prey models incorporating Holling type-I and II functional responses, influenced by this new Allee function. A rich dynamics shows up in the presence of the said function, including the emergence of the limit cycles through the Hopf bifurcation for a particular parameter domain.
Deterministic Global Optimization of the Acquisition Function in Bayesian Optimization: To Do or Not To Do?
Authors: Anastasia Georgiou, Daniel Jungen, Luise Kaven, Verena Hunstig, Constantine Frangakis, Ioannis Kevrekidis, Alexander Mitsos
Date: 2025-03-05 00:00:00
Tasks: Bayesian Optimization, CPU, Gaussian Processes, global-optimization

Bayesian Optimization (BO) with Gaussian Processes relies on optimizing an acquisition function to determine sampling. We investigate the advantages and disadvantages of using a deterministic global solver (MAiNGO) compared to conventional local and stochastic global solvers (L-BFGS-B and multi-start, respectively) for the optimization of the acquisition function. For CPU efficiency, we set a time limit for MAiNGO, taking the best point as optimal. We perform repeated numerical experiments, initially using the Muller-Brown potential as a benchmark function, utilizing the lower confidence bound acquisition function; we further validate our findings with three alternative benchmark functions. Statistical analysis reveals that when the acquisition function is more exploitative (as opposed to exploratory), BO with MAiNGO converges in fewer iterations than with the local solvers. However, when the dataset lacks diversity, or when the acquisition function is overly exploitative, BO with MAiNGO, compared to the local solvers, is more likely to converge to a local rather than a global ly near-optimal solution of the black-box function. L-BFGS-B and multi-start mitigate this risk in BO by introducing stochasticity in the selection of the next sampling point, which enhances the exploration of uncharted regions in the search space and reduces dependence on acquisition function hyperparameters. Ultimately, suboptimal optimization of poorly chosen acquisition functions may be preferable to their optimal solution. When the acquisition function is more exploratory, BO with MAiNGO, multi-start, and L-BFGS-B achieve comparable probabilities of convergence to a globally near-optimal solution (although BO with MAiNGO may require more iterations to converge under these conditions).
The Distributionally Robust Optimization Model of Sparse Principal Component Analysis
Authors: Lei Wang, Xin Liu, Xiaojun Chen
Date: 2025-03-04 00:00:00
Tasks: Riemannian optimization

We consider sparse principal component analysis (PCA) under a stochastic setting where the underlying probability distribution of the random parameter is uncertain. This problem is formulated as a distributionally robust optimization (DRO) model based on a constructive approach to capturing uncertainty in the covariance matrix, which constitutes a nonsmooth constrained min-max optimization problem. We further prove that the inner maximization problem admits a closed-form solution, reformulating the original DRO model into an equivalent minimization problem on the Stiefel manifold. This transformation leads to a Riemannian optimization problem with intricate nonsmooth terms, a challenging formulation beyond the reach of existing algorithms. To address this issue, we devise an efficient smoothing manifold proximal gradient algorithm. We prove the Riemannian gradient consistency and global convergence of our algorithm to a stationary point of the nonsmooth minimization problem. Moreover, we establish the iteration complexity of our algorithm. Finally, numerical experiments are conducted to validate the effectiveness and scalability of our algorithm, as well as to highlight the necessity and rationality of adopting the DRO model for sparse PCA.
Sharpness-Aware Minimization: General Analysis and Improved Rates
Authors: Dimitris Oikonomou, Nicolas Loizou
Date: 2025-03-04 00:00:00
Tasks: image-classification, Image Classification

Sharpness-Aware Minimization (SAM) has emerged as a powerful method for improving generalization in machine learning models by minimizing the sharpness of the loss landscape. However, despite its success, several important questions regarding the convergence properties of SAM in non-convex settings are still open, including the benefits of using normalization in the update rule, the dependence of the analysis on the restrictive bounded variance assumption, and the convergence guarantees under different sampling strategies. To address these questions, in this paper, we provide a unified analysis of SAM and its unnormalized variant (USAM) under one single flexible update rule (Unified SAM), and we present convergence results of the new algorithm under a relaxed and more natural assumption on the stochastic noise. Our analysis provides convergence guarantees for SAM under different step size selections for non-convex problems and functions that satisfy the Polyak-Lojasiewicz (PL) condition (a non-convex generalization of strongly convex functions). The proposed theory holds under the arbitrary sampling paradigm, which includes importance sampling as special case, allowing us to analyze variants of SAM that were never explicitly considered in the literature. Experiments validate the theoretical findings and further demonstrate the practical effectiveness of Unified SAM in training deep neural networks for image classification tasks.
From Data to Uncertainty Sets: a Machine Learning Approach
Authors: Dimitris Bertsimas, Benjamin Boucher
Date: 2025-03-04 00:00:00

Existing approaches of prescriptive analytics -- where inputs of an optimization model can be predicted by leveraging covariates in a machine learning model -- often attempt to optimize the mean value of an uncertain objective. However, when applied to uncertain constraints, these methods rarely work because satisfying a crucial constraint in expectation may result in a high probability of violation. To remedy this, we leverage robust optimization to protect a constraint against the uncertainty of a machine learning model's output. To do so, we design an uncertainty set based on the model's loss function. Intuitively, this approach attempts to minimize the uncertainty around a prediction. Extending guarantees from the robust optimization literature, we derive strong guarantees on the probability of violation. On synthetic computational experiments, our method requires uncertainty sets with radii up to one order of magnitude smaller than those of other approaches.
Is Bellman Equation Enough for Learning Control?
Authors: Haoxiang You, Lekan Molu, Ian Abraham
Date: 2025-03-04 00:00:00

The Bellman equation and its continuous-time counterpart, the Hamilton-Jacobi-Bellman (HJB) equation, serve as necessary conditions for optimality in reinforcement learning and optimal control. While the value function is known to be the unique solution to the Bellman equation in tabular settings, we demonstrate that this uniqueness fails to hold in continuous state spaces. Specifically, for linear dynamical systems, we prove the Bellman equation admits at least $\binom{2n}{n}$ solutions, where $n$ is the state dimension. Crucially, only one of these solutions yields both an optimal policy and a stable closed-loop system. We then demonstrate a common failure mode in value-based methods: convergence to unstable solutions due to the exponential imbalance between admissible and inadmissible solutions. Finally, we introduce a positive-definite neural architecture that guarantees convergence to the stable solution by construction to address this issue.
Probabilistic Insights for Efficient Exploration Strategies in Reinforcement Learning
Authors: Ernesto Garcia, Paola Bermolen, Matthieu Jonckheere, Seva Shneer
Date: 2025-03-05 00:00:00
Tasks: Diversity, Efficient Exploration, reinforcement-learning, Reinforcement Learning

We investigate efficient exploration strategies of environments with unknown stochastic dynamics and sparse rewards. Specifically, we analyze first the impact of parallel simulations on the probability of reaching rare states within a finite time budget. Using simplified models based on random walks and L\'evy processes, we provide analytical results that demonstrate a phase transition in reaching probabilities as a function of the number of parallel simulations. We identify an optimal number of parallel simulations that balances exploration diversity and time allocation. Additionally, we analyze a restarting mechanism that exponentially enhances the probability of success by redirecting efforts toward more promising regions of the state space. Our findings contribute to a more qualitative and quantitative theory of some exploration schemes in reinforcement learning, offering insights into developing more efficient strategies for environments characterized by rare events.
A Criterion for Extending Continuous-Mixture Identifiability Results
Authors: Michael R. Powers, Jiaxin Xu
Date: 2025-03-05 00:00:00

Mixture distributions provide a versatile and widely used framework for modeling random phenomena, and are particularly well-suited to the analysis of geoscientific processes and their attendant risks to society. For continuous mixtures of random variables, we specify a simple criterion - generating-function accessibility - to extend previously known kernel-based identifiability (or unidentifiability) results to new kernel distributions. This criterion, based on functional relationships between the relevant kernels' moment-generating functions or Laplace transforms, may be applied to continuous mixtures of both discrete and continuous random variables. To illustrate the proposed approach, we present results for several specific kernels, in each case briefly noting its relevance to research in the geosciences and/or related risk analysis.
Statistical Limits in Random Tensors with Multiple Correlated Spikes
Authors: Yang Qi, Alexis Decurninge
Date: 2025-03-05 00:00:00

We use tools from random matrix theory to study the multi-spiked tensor model, i.e., a rank-$r$ deformation of a symmetric random Gaussian tensor. In particular, thanks to the nature of local optimization methods used to find the maximum likelihood estimator of this model, we propose to study the phase transition phenomenon for finding critical points of the corresponding optimization problem, i.e., those points defined by the Karush-Kuhn-Tucker (KKT) conditions. Moreover, we characterize the limiting alignments between the estimated signals corresponding to a critical point of the likelihood and the ground truth signals. With the help of these results, we propose a new estimator of the rank-$r$ tensor weights by solving a system of polynomial equations, which is asymptotically unbiased contrary the maximum likelihood estimator.
Active operator learning with predictive uncertainty quantification for partial differential equations
Authors: Nick Winovich, Mitchell Daneker, Lu Lu, Guang Lin
Date: 2025-03-05 00:00:00
Tasks: Active Learning, Operator learning, Uncertainty Quantification

In this work, we develop a method for uncertainty quantification in deep operator networks (DeepONets) using predictive uncertainty estimates calibrated to model errors observed during training. The uncertainty framework operates using a single network, in contrast to existing ensemble approaches, and introduces minimal overhead during training and inference. We also introduce an optimized implementation for DeepONet inference (reducing evaluation times by a factor of five) to provide models well-suited for real-time applications. We evaluate the uncertainty-equipped models on a series of partial differential equation (PDE) problems, and show that the model predictions are unbiased, non-skewed, and accurately reproduce solutions to the PDEs. To assess how well the models generalize, we evaluate the network predictions and uncertainty estimates on in-distribution and out-of-distribution test datasets. We find the predictive uncertainties accurately reflect the observed model errors over a range of problems with varying complexity; simpler out-of-distribution examples are assigned low uncertainty estimates, consistent with the observed errors, while more complex out-of-distribution examples are properly assigned higher uncertainties. We also provide a statistical analysis of the predictive uncertainties and verify that these estimates are well-aligned with the observed error distributions at the tail-end of training. Finally, we demonstrate how predictive uncertainties can be used within an active learning framework to yield improvements in accuracy and data-efficiency for outer-loop optimization procedures.
Applications of Entropy in Data Analysis and Machine Learning: A Review
Authors: Salomé A. Sepúveda Fontaine, José M. Amigó
Date: 2025-03-04 00:00:00

Since its origin in the thermodynamics of the 19th century, the concept of entropy has also permeated other fields of physics and mathematics, such as Classical and Quantum Statistical Mechanics, Information Theory, Probability Theory, Ergodic Theory and the Theory of Dynamical Systems. Specifically, we are referring to the classical entropies: the Boltzmann-Gibbs, von Neumann, Shannon, Kolmogorov-Sinai and topological entropies. In addition to their common name, which is historically justified (as we briefly describe in this review), other commonality of the classical entropies is the important role that they have played and are still playing in the theory and applications of their respective fields and beyond. Therefore, it is not surprising that, in the course of time, many other instances of the overarching concept of entropy have been proposed, most of them tailored to specific purposes. Following the current usage, we will refer to all of them, whether classical or new, simply as entropies. Precisely, the subject of this review is their applications in data analysis and machine learning. The reason for these particular applications is that entropies are very well suited to characterize probability mass distributions, typically generated by finite-state processes or symbolized signals. Therefore, we will focus on entropies defined as positive functionals on probability mass distributions and provide an axiomatic characterization that goes back to Shannon and Khinchin. Given the plethora of entropies in the literature, we have selected a representative group, including the classical ones. The applications summarized in this review finely illustrate the power and versatility of entropy in data analysis and machine learning.
Validity of the total quasi-steady-state approximation in stochastic biochemical reaction networks
Authors: Yun Min Song, Kangmin Lee, Jae Kyoung Kim
Date: 2025-03-02 00:00:00
Tasks: valid

Stochastic models for biochemical reaction networks are widely used to explore their complex dynamics but face significant challenges, including difficulties in determining rate constants and high computational costs. To address these issues, model reduction approaches based on deterministic quasi-steady-state approximations (QSSA) have been employed, resulting in propensity functions in the form of deterministic non-elementary reaction functions, such as the Michaelis-Menten equation. In particular, the total QSSA (tQSSA), known for its accuracy in deterministic frameworks, has been perceived as universally valid for stochastic model reduction. However, recent studies have challenged this perception. In this review, we demonstrate that applying tQSSA in stochastic model reduction can distort dynamics, even in cases where the deterministic tQSSA is rigorously valid. This highlights the need for caution when using deterministic QSSA in stochastic model reduction to avoid erroneous conclusions from model simulations.
Understanding the Commodity Futures Term Structure Through Signatures
Authors: Hari P. Krishnan, Stephan Sturm
Date: 2025-03-01 00:00:00

Signature methods have been widely and effectively used as a tool for feature extraction in statistical learning methods, notably in mathematical finance. They lack, however, interpretability: in the general case, it is unclear why signatures actually work. The present article aims to address this issue directly, by introducing and developing the concept of signature perturbations. In particular, we construct a regular perturbation of the signature of the term structure of log prices for various commodities, in terms of the convenience yield. Our perturbation expansion and rigorous convergence estimates help explain the success of signature-based classification of commodities markets according to their term structure, with the volatility of the convenience yield as the major discriminant.
Early-Stopped Mirror Descent for Linear Regression over Convex Bodies
Authors: Tobias Wegel, Gil Kur, Patrick Rebeschini
Date: 2025-03-05 00:00:00
Tasks: regression

Early-stopped iterative optimization methods are widely used as alternatives to explicit regularization, and direct comparisons between early-stopping and explicit regularization have been established for many optimization geometries. However, most analyses depend heavily on the specific properties of the optimization geometry or strong convexity of the empirical objective, and it remains unclear whether early-stopping could ever be less statistically efficient than explicit regularization for some particular shape constraint, especially in the overparameterized regime. To address this question, we study the setting of high-dimensional linear regression under additive Gaussian noise when the ground truth is assumed to lie in a known convex body and the task is to minimize the in-sample mean squared error. Our main result shows that for any convex body and any design matrix, up to an absolute constant factor, the worst-case risk of unconstrained early-stopped mirror descent with an appropriate potential is at most that of the least squares estimator constrained to the convex body. We achieve this by constructing algorithmic regularizers based on the Minkowski functional of the convex body.
An Analytical Theory of Power Law Spectral Bias in the Learning Dynamics of Diffusion Models
Authors: Binxu Wang
Date: 2025-03-05 00:00:00

We developed an analytical framework for understanding how the learned distribution evolves during diffusion model training. Leveraging the Gaussian equivalence principle, we derived exact solutions for the gradient-flow dynamics of weights in one- or two-layer linear denoiser settings with arbitrary data. Remarkably, these solutions allowed us to derive the generated distribution in closed form and its KL divergence through training. These analytical results expose a pronounced power-law spectral bias, i.e., for weights and distributions, the convergence time of a mode follows an inverse power law of its variance. Empirical experiments on both Gaussian and image datasets demonstrate that the power-law spectral bias remains robust even when using deeper or convolutional architectures. Our results underscore the importance of the data covariance in dictating the order and rate at which diffusion models learn different modes of the data, providing potential explanations for why earlier stopping could lead to incorrect details in image generative models.
A Near Complete Nonasymptotic Generalization Theory For Multilayer Neural Networks: Beyond the Bias-Variance Tradeoff
Authors: Hao Yu, Xiangyang Ji
Date: 2025-03-03 00:00:00

We propose a first near complete (that will make explicit sense in the main text) nonasymptotic generalization theory for multilayer neural networks with arbitrary Lipschitz activations and general Lipschitz loss functions (with some very mild conditions). In particular, it doens't require the boundness of loss function, as commonly assumed in the literature. Our theory goes beyond the bias-variance tradeoff, aligned with phenomenon typically encountered in deep learning. It is therefore sharp different with other existing nonasymptotic generalization error bounds for neural networks. More explicitly, we propose an explicit generalization error upper bound for multilayer neural networks with arbitrary Lipschitz activations $\sigma$ with $\sigma(0)=0$ and broad enough Lipschitz loss functions, without requiring either the width, depth or other hyperparameters of the neural network approaching infinity, a specific neural network architect (e.g. sparsity, boundness of some norms), a particular activation function, a particular optimization algorithm or boundness of the loss function, and with taking the approximation error into consideration. General Lipschitz activation can also be accommodated into our framework. A feature of our theory is that it also considers approximation errors. Furthermore, we show the near minimax optimality of our theory for multilayer ReLU networks for regression problems. Notably, our upper bound exhibits the famous double descent phenomenon for such networks, which is the most distinguished characteristic compared with other existing results. This work emphasizes a view that many classical results should be improved to embrace the unintuitive characteristics of deep learning to get a better understanding of it.