pwc-archive/papers-with-abstracts huggingface dataset:
Fast Derivative Valuation from Volatility Surfaces using Machine Learning
Authors: Lijie Ding, Egang Lu, Kin Cheung
Date: 2025-05-29 00:00:00
Tasks: GPR

We introduce a fast and flexible Machine Learning (ML) framework for pricing derivative products whose valuation depends on volatility surfaces. By parameterizing volatility surfaces with the 5-parameter stochastic volatility inspired (SVI) model augmented by a one-factor term structure adjustment, we first generate numerous volatility surfaces over realistic ranges for these parameters. From these synthetic market scenarios, we then compute high-accuracy valuations using conventional methodologies for two representative products: the fair strike of a variance swap and the price and Greeks of an American put. We then train the Gaussian Process Regressor (GPR) to learn the nonlinear mapping from the input risk factors, which are the volatility surface parameters, strike and interest rate, to the valuation outputs. Once trained, We use the GPR to perform out-of-sample valuations and compare the results against valuations using conventional methodologies. Our ML model achieves very accurate results of $0.5\%$ relative error for the fair strike of variance swap and $1.7\% \sim 3.5\%$ relative error for American put prices and first-order Greeks. More importantly, after training, the model computes valuations almost instantly, yielding a three to four orders of magnitude speedup over Crank-Nicolson finite-difference method for American puts, enabling real-time risk analytics, dynamic hedging and large-scale scenario analysis. Our approach is general and can be extended to other path-dependent derivative products with early-exercise features, paving the way for hybrid quantitative engines for modern financial systems.
The Role of Intangible Investment in Predicting Stock Returns: Six Decades of Evidence
Authors: Lin Li
Date: 2025-05-22 00:00:00

Using an intangible intensity factor that is orthogonal to the Fama--French factors, we compare the role of intangible investment in predicting stock returns over the periods 1963--1992 and 1993--2022. For 1963--1992, intangible investment is weak in predicting stock returns, but for 1993--2022, the predictive power of intangible investment becomes very strong. Intangible investment has a significant impact not only on the MTB ratio (Fama--French high minus low [HML] factor) but also on operating profitability (OP) (Fama--French robust minus weak [RMW] factor) when forecasting stock returns from 1993 to 2022. For intangible asset-intensive firms, intangible investment is the main predictor of stock returns, rather than MTB ratio and profitability. Our evidence suggests that intangible investment has become an important factor in explaining stock returns over time, independent of other factors such as profitability and MTB ratio.
Replication of Reference-Dependent Preferences and the Risk-Return Trade-Off in the Chinese Market
Authors: Penggan Xu
Date: 2025-05-27 00:00:00

This study replicates the findings of Wang et al. (2017) on reference-dependent preferences and their impact on the risk-return trade-off in the Chinese stock market, a unique context characterized by high retail investor participation, speculative trading behavior, and regulatory complexities. Capital Gains Overhang (CGO), a proxy for unrealized gains or losses, is employed to explore how behavioral biases shape cross-sectional stock returns in an emerging market setting. Utilizing data from 1995 to 2024 and econometric techniques such as Dependent Double Sorting and Fama-MacBeth regressions, this research investigates the interaction between CGO and five risk proxies: Beta, Return Volatility (RETVOL), Idiosyncratic Volatility (IVOL), Firm Age (AGE), and Cash Flow Volatility (CFVOL). Key findings reveal a weaker or absent positive risk-return relationship among high-CGO firms and stronger positive relationships among low-CGO firms, diverging from U.S. market results, and the interaction effects between CGO and risk proxies, significant and positive in the U.S., are predominantly negative in the Chinese market, reflecting structural and behavioral differences, such as speculative trading and diminished reliance on reference points. The results suggest that reference-dependent preferences play a less pronounced role in the Chinese market, emphasizing the need for tailored investment strategies in emerging economies.
Hybrid Models for Financial Forecasting: Combining Econometric, Machine Learning, and Deep Learning Models
Authors: Dominik Stempień, Robert Ślepaczuk
Date: 2025-05-26 00:00:00

This research systematically develops and evaluates various hybrid modeling approaches by combining traditional econometric models (ARIMA and ARFIMA models) with machine learning and deep learning techniques (SVM, XGBoost, and LSTM models) to forecast financial time series. The empirical analysis is based on two distinct financial assets: the S&P 500 index and Bitcoin. By incorporating over two decades of daily data for the S&P 500 and almost ten years of Bitcoin data, the study provides a comprehensive evaluation of forecasting methodologies across different market conditions and periods of financial distress. Models' training and hyperparameter tuning procedure is performed using a novel three-fold dynamic cross-validation method. The applicability of applied models is evaluated using both forecast error metrics and trading performance indicators. The obtained findings indicate that the proper construction process of hybrid models plays a crucial role in developing profitable trading strategies, outperforming their individual components and the benchmark Buy&Hold strategy. The most effective hybrid model architecture was achieved by combining the econometric ARIMA model with either SVM or LSTM, under the assumption of a non-additive relationship between the linear and nonlinear components.
Agent-based Liquidity Risk Modelling for Financial Markets
Authors: Perukrishnen Vytelingum, Rory Baggott, Namid Stillman, Jianfei Zhang, Dingqiu Zhu, Tao Chen, Justin Lyon
Date: 2025-05-21 00:00:00

In this paper, we describe a novel agent-based approach for modelling the transaction cost of buying or selling an asset in financial markets, e.g., to liquidate a large position as a result of a margin call to meet financial obligations. The simple act of buying or selling in the market causes a price impact and there is a cost described as liquidity risk. For example, when selling a large order, there is market slippage -- each successive trade will execute at the same or worse price. When the market adjusts to the new information revealed by the execution of such a large order, we observe in the data a permanent price impact that can be attributed to the change in the fundamental value as market participants reassess the value of the asset. In our ABM model, we introduce a novel mechanism where traders assume orderflow is informed and each trade reveals some information about the value of the asset, and traders update their belief of the fundamental value for every trade. The result is emergent, realistic price impact without oversimplifying the problem as most stylised models do, but within a realistic framework that models the exchange with its protocols, its limit orderbook and its auction mechanism and that can calculate the transaction cost of any execution strategy without limitation. Our stochastic ABM model calculates the costs and uncertainties of buying and selling in a market by running Monte-Carlo simulations, for a better understanding of liquidity risk and can be used to optimise for optimal execution under liquidity risk. We demonstrate its practical application in the real world by calculating the liquidity risk for the Hang-Seng Futures Index.
An Efficient deep learning model to Predict Stock Price Movement Based on Limit Order Book
Authors: Jiahao Yang, Ran Fang, Ming Zhang, Jun Zhou
Date: 2025-05-14 00:00:00

In high-frequency trading (HFT), leveraging limit order books (LOB) to model stock price movements is crucial for achieving profitable outcomes. However, this task is challenging due to the high-dimensional and volatile nature of the original data. Even recent deep learning models often struggle to capture price movement patterns effectively, particularly without well-designed features. We observed that raw LOB data exhibits inherent symmetry between the ask and bid sides, and the bid-ask differences demonstrate greater stability and lower complexity compared to the original data. Building on this insight, we propose a novel approach in which leverages the Siamese architecture to enhance the performance of existing deep learning models. The core idea involves processing the ask and bid sides separately using the same module with shared parameters. We applied our Siamese-based methods to several widely used strong baselines and validated their effectiveness using data from 14 military industry stocks in the Chinese A-share market. Furthermore, we integrated multi-head attention (MHA) mechanisms with the Long Short-Term Memory (LSTM) module to investigate its role in modeling stock price movements. Our experiments used raw data and widely used Order Flow Imbalance (OFI) features as input with some strong baseline models. The results show that our method improves the performance of strong baselines in over 75$% of cases, excluding the Multi-Layer Perception (MLP) baseline, which performed poorly and is not considered practical. Furthermore, we found that Multi-Head Attention can enhance model performance, particularly over shorter forecasting horizons.
Deep Learning-Based BMD Estimation from Radiographs with Conformal Uncertainty Quantification
Authors: Long Hui, Wai Lok Yeung
Date: 2025-05-28 00:00:00
Tasks: Conformal Prediction, Prediction Intervals, Uncertainty Quantification

Limited DXA access hinders osteoporosis screening. This proof-of-concept study proposes using widely available knee X-rays for opportunistic Bone Mineral Density (BMD) estimation via deep learning, emphasizing robust uncertainty quantification essential for clinical use. An EfficientNet model was trained on the OAI dataset to predict BMD from bilateral knee radiographs. Two Test-Time Augmentation (TTA) methods were compared: traditional averaging and a multi-sample approach. Crucially, Split Conformal Prediction was implemented to provide statistically rigorous, patient-specific prediction intervals with guaranteed coverage. Results showed a Pearson correlation of 0.68 (traditional TTA). While traditional TTA yielded better point predictions, the multi-sample approach produced slightly tighter confidence intervals (90%, 95%, 99%) while maintaining coverage. The framework appropriately expressed higher uncertainty for challenging cases. Although anatomical mismatch between knee X-rays and standard DXA limits immediate clinical use, this method establishes a foundation for trustworthy AI-assisted BMD screening using routine radiographs, potentially improving early osteoporosis detection.
Unsupervised Latent Pattern Analysis for Estimating Type 2 Diabetes Risk in Undiagnosed Populations
Authors: Praveen Kumar, Vincent T. Metzger, Scott A. Malec
Date: 2025-05-27 00:00:00

The global prevalence of diabetes, particularly type 2 diabetes mellitus (T2DM), is rapidly increasing, posing significant health and economic challenges. T2DM not only disrupts blood glucose regulation but also damages vital organs such as the heart, kidneys, eyes, nerves, and blood vessels, leading to substantial morbidity and mortality. In the US alone, the economic burden of diagnosed diabetes exceeded \$400 billion in 2022. Early detection of individuals at risk is critical to mitigating these impacts. While machine learning approaches for T2DM prediction are increasingly adopted, many rely on supervised learning, which is often limited by the lack of confirmed negative cases. To address this limitation, we propose a novel unsupervised framework that integrates Non-negative Matrix Factorization (NMF) with statistical techniques to identify individuals at risk of developing T2DM. Our method identifies latent patterns of multimorbidity and polypharmacy among diagnosed T2DM patients and applies these patterns to estimate the T2DM risk in undiagnosed individuals. By leveraging data-driven insights from comorbidity and medication usage, our approach provides an interpretable and scalable solution that can assist healthcare providers in implementing timely interventions, ultimately improving patient outcomes and potentially reducing the future health and economic burden of T2DM.
pared: Model selection using multi-objective optimization
Authors: Priyam Das, Sarah Robinson, Christine B. Peterson
Date: 2025-05-27 00:00:00
Tasks: model, Model Selection

Motivation: Model selection is a ubiquitous challenge in statistics. For penalized models, model selection typically entails tuning hyperparameters to maximize a measure of fit or minimize out-of-sample prediction error. However, these criteria fail to reflect other desirable characteristics, such as model sparsity, interpretability, or smoothness. Results: We present the R package pared to enable the use of multi-objective optimization for model selection. Our approach entails the use of Gaussian process-based optimization to efficiently identify solutions that represent desirable trade-offs. Our implementation includes popular models with multiple objectives including the elastic net, fused lasso, fused graphical lasso, and group graphical lasso. Our R package generates interactive graphics that allow the user to identify hyperparameter values that result in fitted models which lie on the Pareto frontier. Availability: We provide the R package pared and vignettes illustrating its application to both simulated and real data at https://github.com/priyamdas2/pared.
Generating Hypotheses of Dynamic Causal Graphs in Neuroscience: Leveraging Generative Factor Models of Observed Time Series
Authors: Zachary C. Brown, David Carlson
Date: 2025-05-27 00:00:00
Tasks: Causal Discovery, Time Series

The field of hypothesis generation promises to reduce costs in neuroscience by narrowing the range of interventional studies needed to study various phenomena. Existing machine learning methods can generate scientific hypotheses from complex datasets, but many approaches assume causal relationships are static over time, limiting their applicability to systems with dynamic, state-dependent behavior, such as the brain. While some techniques attempt dynamic causal discovery through factor models, they often restrict relationships to linear patterns or impose other simplifying assumptions. We propose a novel method that models dynamic graphs as a conditionally weighted superposition of static graphs, where each static graph can capture nonlinear relationships. This approach enables the detection of complex, time-varying interactions between variables beyond linear limitations. Our method improves f1-scores of predicted dynamic causal patterns by roughly 22-28% on average over baselines in some of our experiments, with some improvements reaching well over 60%. A case study on real brain data demonstrates our method's ability to uncover relationships linked to specific behavioral states, offering valuable insights into neural dynamics.
Robust Distributed Estimation: Extending Gossip Algorithms to Ranking and Trimmed Means
Authors: Anna van Elst, Igor Colin, Stephan Clémençon
Date: 2025-05-23 00:00:00

This paper addresses the problem of robust estimation in gossip algorithms over arbitrary communication graphs. Gossip algorithms are fully decentralized, relying only on local neighbor-to-neighbor communication, making them well-suited for situations where communication is constrained. A fundamental challenge in existing mean-based gossip algorithms is their vulnerability to malicious or corrupted nodes. In this paper, we show that an outlier-robust mean can be computed by globally estimating a robust statistic. More specifically, we propose a novel gossip algorithm for rank estimation, referred to as \textsc{GoRank}, and leverage it to design a gossip procedure dedicated to trimmed mean estimation, coined \textsc{GoTrim}. In addition to a detailed description of the proposed methods, a key contribution of our work is a precise convergence analysis: we establish an $\mathcal{O}(1/t)$ rate for rank estimation and an $\mathcal{O}((\log t)/\sqrt{t})$ rate for trimmed mean estimation, where by $t$ is meant the number of iterations. Moreover, we provide a breakdown point analysis of \textsc{GoTrim}. We empirically validate our theoretical results through experiments on diverse network topologies, data distributions and contamination schemes.
Transfer Faster, Price Smarter: Minimax Dynamic Pricing under Cross-Market Preference Shift
Authors: Yi Zhang, Elynn Chen, Yujun Yan
Date: 2025-05-22 00:00:00
Tasks: Transfer Learning

We study contextual dynamic pricing when a target market can leverage K auxiliary markets -- offline logs or concurrent streams -- whose mean utilities differ by a structured preference shift. We propose Cross-Market Transfer Dynamic Pricing (CM-TDP), the first algorithm that provably handles such model-shift transfer and delivers minimax-optimal regret for both linear and non-parametric utility models. For linear utilities of dimension d, where the difference between source- and target-task coefficients is $s_{0}$-sparse, CM-TDP attains regret $\tilde{O}((d*K^{-1}+s_{0})\log T)$. For nonlinear demand residing in a reproducing kernel Hilbert space with effective dimension $\alpha$, complexity $\beta$ and task-similarity parameter $H$, the regret becomes $\tilde{O}\!(K^{-2\alpha\beta/(2\alpha\beta+1)}T^{1/(2\alpha\beta+1)} + H^{2/(2\alpha+1)}T^{1/(2\alpha+1)})$, matching information-theoretic lower bounds up to logarithmic factors. The RKHS bound is the first of its kind for transfer pricing and is of independent interest. Extensive simulations show up to 50% lower cumulative regret and 5 times faster learning relative to single-market pricing baselines. By bridging transfer learning, robust aggregation, and revenue optimization, CM-TDP moves toward pricing systems that transfer faster, price smarter.
Are Statistical Methods Obsolete in the Era of Deep Learning?
Authors: Skyler Wu, Shihao Yang, S. C. Kou
Date: 2025-05-27 00:00:00
Tasks: Deep Learning, Epidemiology, Future prediction

In the era of AI, neural networks have become increasingly popular for modeling, inference, and prediction, largely due to their potential for universal approximation. With the proliferation of such deep learning models, a question arises: are leaner statistical methods still relevant? To shed insight on this question, we employ the mechanistic nonlinear ordinary differential equation (ODE) inverse problem as a testbed, using physics-informed neural network (PINN) as a representative of the deep learning paradigm and manifold-constrained Gaussian process inference (MAGI) as a representative of statistically principled methods. Through case studies involving the SEIR model from epidemiology and the Lorenz model from chaotic dynamics, we demonstrate that statistical methods are far from obsolete, especially when working with sparse and noisy observations. On tasks such as parameter inference and trajectory reconstruction, statistically principled methods consistently achieve lower bias and variance, while using far fewer parameters and requiring less hyperparameter tuning. Statistical methods can also decisively outperform deep learning models on out-of-sample future prediction, where the absence of relevant data often leads overparameterized models astray. Additionally, we find that statistically principled approaches are more robust to accumulation of numerical imprecision and can represent the underlying system more faithful to the true governing ODEs.
tenSVD algorithm for compression
Authors: Michele Gallo
Date: 2025-05-27 00:00:00

Tensors provide a robust framework for managing high-dimensional data. Consequently, tensor analysis has emerged as an active research area in various domains, including machine learning, signal processing, computer vision, graph analysis, and data mining. This study introduces an efficient image storage approach utilizing tensors, aiming to minimize memory to store, bandwidth to transmit and energy to processing. The proposed method organizes original data into a higher-order tensor and applies the Tucker model for compression. Implemented in R, this method is compared to a baseline algorithm. The evaluation focuses on efficient of algorithm measured in term of computational time and the quality of information preserved, using both simulated and real datasets. A detailed analysis of the results is conducted, employing established quantitative metrics, with significant attention paid to sustainability in terms of energy consumption across algorithms.
Stability Selection via Variable Decorrelation
Authors: Mahdi Nouraie, Connor Smith, Samuel Muller
Date: 2025-05-27 00:00:00
Tasks: Variable Selection

The Lasso is a prominent algorithm for variable selection. However, its instability in the presence of correlated variables in the high-dimensional setting is well-documented. Although previous research has attempted to address this issue by modifying the Lasso loss function, this paper introduces an approach that simplifies the data processed by Lasso. We propose that decorrelating variables before applying the Lasso improves the stability of variable selection regardless of the direction of correlation among predictors. Furthermore, we highlight that the irrepresentable condition, which ensures consistency for the Lasso, is satisfied after variable decorrelation under two assumptions. In addition, by noting that the instability of the Lasso is not limited to high-dimensional settings, we demonstrate the effectiveness of the proposed approach for low-dimensional data. Finally, we present empirical results that indicate the efficacy of the proposed method across different variable selection techniques, highlighting its potential for broader application. The DVS R package is developed to facilitate the implementation of the methodology proposed in this paper.
Statistical Inference for Online Algorithms
Authors: Selina Carter, Arun K Kuchibhotla
Date: 2025-05-22 00:00:00
Tasks: valid

Construction of confidence intervals and hypothesis tests for functionals based on asymptotically normal estimators is a classical topic in statistical inference. The simplest and in many cases optimal inference procedure is the Wald interval or the likelihood ratio test, both of which require an estimator and an estimate of the asymptotic variance of the estimator. Estimators obtained from online/sequential algorithms forces one to consider the computational aspects of the inference problem, i.e., one cannot access all of the data as many times as needed. Several works on this topic explored the online estimation of asymptotic variance. In this article, we propose computationally efficient, rate-optimal, and asymptotically valid confidence regions based on the output of online algorithms {\em without} estimating the asymptotic variance. As a special case, this implies inference from any algorithm that yields an asymptotically normal estimator. We focus our efforts on stochastic gradient descent with Polyak averaging to understand the practical performance of the proposed method.