pwc-archive/papers-with-abstracts huggingface dataset:
On the Interpolation Effect of Score Smoothing
Authors: Zhengdao Chen
Date: 2025-02-26 00:00:00
Tasks: Denoising, Memorization

Score-based diffusion models have achieved remarkable progress in various domains with the ability to generate new data samples that do not exist in the training set. In this work, we examine the hypothesis that their generalization ability arises from an interpolation effect caused by a smoothing of the empirical score function. Focusing on settings where the training set lies uniformly in a one-dimensional linear subspace, we study the interplay between score smoothing and the denoising dynamics with mathematically solvable models. In particular, we demonstrate how a smoothed score function can lead to the generation of samples that interpolate among the training data within their subspace while avoiding full memorization. We also present evidence that learning score functions with regularized neural networks can have a similar effect on the denoising dynamics as score smoothing.
Framework for asset-liability management with fixed-term securities
Authors: Yevhen Havrylenko
Date: 2025-02-26 00:00:00
Tasks: Management

We consider an optimal investment-consumption problem for a utility-maximizing investor who has access to assets with different liquidity and whose consumption rate as well as terminal wealth are subject to lower-bound constraints. Assuming utility functions that satisfy standard conditions, we develop a methodology for deriving the optimal strategies in semi-closed form. Our methodology is based on the generalized martingale approach and the decomposition of the problem into subproblems. We illustrate our approach by deriving explicit formulas for agents with power-utility functions and discuss potential extensions of the proposed framework. In numerical studies, we substantiate how the parameters of our framework impact the optimal proportion of initial capital allocated to the illiquid asset, the monetary value that the investor subjectively assigns to the fixed-term asset, and the potential of the illiquid asset to increase terminal the terminal value of liabilities without loss in the investor's expected utility.
Langevin Multiplicative Weights Update with Applications in Polynomial Portfolio Management
Authors: Yi Feng, Xiao Wang, Tian Xie
Date: 2025-02-26 00:00:00
Tasks: global-optimization, Management

We consider nonconvex optimization problem over simplex, and more generally, a product of simplices. We provide an algorithm, Langevin Multiplicative Weights Update (LMWU) for solving global optimization problems by adding a noise scaling with the non-Euclidean geometry in the simplex. Non-convex optimization has been extensively studied by machine learning community due to its application in various scenarios such as neural network approximation and finding Nash equilibrium. Despite recent progresses on provable guarantee of escaping and avoiding saddle point (convergence to local minima) and global convergence of Langevin gradient based method without constraints, the global optimization with constraints is less studied. We show that LMWU algorithm is provably convergent to interior global minima with a non-asymptotic convergence analysis. We verify the efficiency of the proposed algorithm in real data set from polynomial portfolio management, where optimization of a highly non-linear objective function plays a crucial role.
The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
Authors: Jinbo Wang, Mingze Wang, Zhanpeng Zhou, Junchi Yan, Weinan E, Lei Wu
Date: 2025-02-26 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model

Transformers consist of diverse building blocks, such as embedding layers, normalization layers, self-attention mechanisms, and point-wise feedforward networks. Thus, understanding the differences and interactions among these blocks is important. In this paper, we uncover a clear Sharpness Disparity across these blocks, which emerges early in training and intriguingly persists throughout the training process. Motivated by this finding, we propose Blockwise Learning Rate (LR), a strategy that tailors the LR to each block's sharpness, accelerating large language model (LLM) pre-training. By integrating Blockwise LR into AdamW, we consistently achieve lower terminal loss and nearly $2\times$ speedup compared to vanilla AdamW. We demonstrate this acceleration across GPT-2 and LLaMA, with model sizes ranging from 0.12B to 2B and datasets of OpenWebText, MiniPile, and C4. Finally, we incorporate Blockwise LR into Adam-mini (Zhang et al., 2024), a recently proposed memory-efficient variant of Adam, achieving a combined $2\times$ speedup and $2\times$ memory saving. These results underscore the potential of exploiting the sharpness disparity to improve LLM training.
Tight Bounds on the Binomial CDF, and the Minimum of i.i.d Binomials, in terms of KL-Divergence
Authors: Xiaohan Zhu, Mesrob I. Ohannessian, Nathan Srebro
Date: 2025-02-25 00:00:00

We provide finite sample upper and lower bounds on the Binomial tail probability which are a direct application of Sanov's theorem. We then use these to obtain high probability upper and lower bounds on the minimum of i.i.d. Binomial random variables. Both bounds are finite sample, asymptotically tight, and expressed in terms of the KL-divergence.
Stein's unbiased risk estimate and Hyvärinen's score matching
Authors: Sulagna Ghosh, Nikolaos Ignatiadis, Frederic Koehler, Amber Lee
Date: 2025-02-27 00:00:00

We study two G-modeling strategies for estimating the signal distribution (the empirical Bayesian's prior) from observations corrupted with normal noise. First, we choose the signal distribution by minimizing Stein's unbiased risk estimate (SURE) of the implied Eddington/Tweedie Bayes denoiser, an approach motivated by optimal empirical Bayesian shrinkage estimation of the signals. Second, we select the signal distribution by minimizing Hyv\"arinen's score matching objective for the implied score (derivative of log-marginal density), targeting minimal Fisher divergence between estimated and true marginal densities. While these strategies appear distinct, they are known to be mathematically equivalent. We provide a unified analysis of SURE and score matching under both well-specified signal distribution classes and misspecification. In the classical well-specified setting with homoscedastic noise and compactly supported signal distribution, we establish nearly parametric rates of convergence of the empirical Bayes regret and the Fisher divergence. In a commonly studied misspecified model, we establish fast rates of convergence to the oracle denoiser and corresponding oracle inequalities. Our empirical results demonstrate competitiveness with nonparametric maximum likelihood in well-specified settings, while showing superior performance under misspecification, particularly in settings involving heteroscedasticity and side information.
Universality of conformal prediction under the assumption of randomness
Authors: Vladimir Vovk
Date: 2025-02-26 00:00:00
Tasks: Conformal Prediction, Prediction, regression, valid

Conformal predictors provide set or functional predictions that are valid under the assumption of randomness, i.e., under the assumption of independent and identically distributed data. The question asked in this paper is whether there are predictors that are valid in the same sense under the assumption of randomness and that are more efficient than conformal predictors. The answer is that the class of conformal predictors is universal in that only limited gains in predictive efficiency are possible. The previous work in this area has relied on the algorithmic theory of randomness and so involved unspecified constants, whereas this paper's results are much more practical. They are also shown to be optimal in some respects.
A Model-Centric Review of Deep Learning for Protein Design
Authors: Gregory W. Kyro, Tianyin Qiu, Victor S. Batista
Date: 2025-02-26 00:00:00
Tasks: Deep Learning, Protein Design, Protein Structure Prediction

Deep learning has transformed protein design, enabling accurate structure prediction, sequence optimization, and de novo protein generation. Advances in single-chain protein structure prediction via AlphaFold2, RoseTTAFold, ESMFold, and others have achieved near-experimental accuracy, inspiring successive work extended to biomolecular complexes via AlphaFold Multimer, RoseTTAFold All-Atom, AlphaFold 3, Chai-1, Boltz-1 and others. Generative models such as ProtGPT2, ProteinMPNN, and RFdiffusion have enabled sequence and backbone design beyond natural evolution-based limitations. More recently, joint sequence-structure co-design models, including ESM3, have integrated both modalities into a unified framework, resulting in improved designability. Despite these advances, challenges still exist pertaining to modeling sequence-structure-function relationships and ensuring robust generalization beyond the regions of protein space spanned by the training data. Future advances will likely focus on joint sequence-structure-function co-design frameworks that are able to model the fitness landscape more effectively than models that treat these modalities independently. Current capabilities, coupled with the dizzying rate of progress, suggest that the field will soon enable rapid, rational design of proteins with tailored structures and functions that transcend the limitations imposed by natural evolution. In this review, we discuss the current capabilities of deep learning methods for protein design, focusing on some of the most revolutionary and capable models with respect to their functionality and the applications that they enable, leading up to the current challenges of the field and the optimal path forward.
SE(3)-Equivariant Ternary Complex Prediction Towards Target Protein Degradation
Authors: Fanglei Xue, Meihan Zhang, Shuqi Li, Xinyu Gao, James A. Wohlschlegel, Wenbing Huang, Yi Yang, Weixian Deng
Date: 2025-02-26 00:00:00
Tasks: Blind Docking, Decoder, Drug Discovery, Graph Attention, Graph Neural Network

Targeted protein degradation (TPD) induced by small molecules has emerged as a rapidly evolving modality in drug discovery, targeting proteins traditionally considered "undruggable". Proteolysis-targeting chimeras (PROTACs) and molecular glue degraders (MGDs) are the primary small molecules that induce TPD. Both types of molecules form a ternary complex linking an E3 ligase with a target protein, a crucial step for drug discovery. While significant advances have been made in binary structure prediction for proteins and small molecules, ternary structure prediction remains challenging due to obscure interaction mechanisms and insufficient training data. Traditional methods relying on manually assigned rules perform poorly and are computationally demanding due to extensive random sampling. In this work, we introduce DeepTernary, a novel deep learning-based approach that directly predicts ternary structures in an end-to-end manner using an encoder-decoder architecture. DeepTernary leverages an SE(3)-equivariant graph neural network (GNN) with both intra-graph and ternary inter-graph attention mechanisms to capture intricate ternary interactions from our collected high-quality training dataset, TernaryDB. The proposed query-based Pocket Points Decoder extracts the 3D structure of the final binding ternary complex from learned ternary embeddings, demonstrating state-of-the-art accuracy and speed in existing PROTAC benchmarks without prior knowledge from known PROTACs. It also achieves notable accuracy on the more challenging MGD benchmark under the blind docking protocol. Remarkably, our experiments reveal that the buried surface area calculated from predicted structures correlates with experimentally obtained degradation potency-related metrics. Consequently, DeepTernary shows potential in effectively assisting and accelerating the development of TPDs for previously undruggable targets.
Towards More Accurate Full-Atom Antibody Co-Design
Authors: Jiayang Wu, Xingyi Zhang, Xiangyu Dong, Kun Xie, Ziqi Liu, Wensheng Gan, Sibo Wang, Le Song
Date: 2025-02-11 00:00:00
Tasks: Specificity

Antibody co-design represents a critical frontier in drug development, where accurate prediction of both 1D sequence and 3D structure of complementarity-determining regions (CDRs) is essential for targeting specific epitopes. Despite recent advances in equivariant graph neural networks for antibody design, current approaches often fall short in capturing the intricate interactions that govern antibody-antigen recognition and binding specificity. In this work, we present Igformer, a novel end-to-end framework that addresses these limitations through innovative modeling of antibody-antigen binding interfaces. Our approach refines the inter-graph representation by integrating personalized propagation with global attention mechanisms, enabling comprehensive capture of the intricate interplay between local chemical interactions and global conformational dependencies that characterize effective antibody-antigen binding. Through extensive validation on epitope-binding CDR design and structure prediction tasks, Igformer demonstrates significant improvements over existing methods, suggesting that explicit modeling of multi-scale residue interactions can substantially advance computational antibody design for therapeutic applications.
Fast and Accurate Antibody Sequence Design via Structure Retrieval
Authors: Xingyi Zhang, Kun Xie, Ningqiao Huang, Wei Liu, Peilin Zhao, Sibo Wang, Kangfei Zhao, Biaobin Jiang
Date: 2025-02-11 00:00:00
Tasks: Graph Neural Network, Protein Design, Retrieval

Recent advancements in protein design have leveraged diffusion models to generate structural scaffolds, followed by a process known as protein inverse folding, which involves sequence inference on these scaffolds. However, these methodologies face significant challenges when applied to hyper-variable structures such as antibody Complementarity-Determining Regions (CDRs), where sequence inference frequently results in non-functional sequences due to hallucinations. Distinguished from prevailing protein inverse folding approaches, this paper introduces Igseek, a novel structure-retrieval framework that infers CDR sequences by retrieving similar structures from a natural antibody database. Specifically, Igseek employs a simple yet effective multi-channel equivariant graph neural network to generate high-quality geometric representations of CDR backbone structures. Subsequently, it aligns sequences of structurally similar CDRs and utilizes structurally conserved sequence motifs to enhance inference accuracy. Our experiments demonstrate that Igseek not only proves to be highly efficient in structural retrieval but also outperforms state-of-the-art approaches in sequence recovery for both antibodies and T-Cell Receptors, offering a new retrieval-based perspective for therapeutic protein design.
Trait-structured chemotaxis: Exploring ligand-receptor dynamics and travelling wave properties in a Keller-Segel model
Authors: Viktoria Freingruber, Tommaso Lorenzi, Kevin J. Painter, Mariya Ptashnyk
Date: 2025-02-26 00:00:00

A novel trait-structured Keller-Segel model that explores the dynamics of a migrating cell population guided by chemotaxis in response to an external ligand concentration is derived and analysed. Unlike traditional Keller-Segel models, this framework introduces an explicit representation of ligand-receptor bindings on the cell membrane, where the percentage of occupied receptors constitutes the trait that influences cellular phenotype. The model posits that the cell's phenotypic state directly modulates its capacity for chemotaxis and proliferation, governed by a trade-off due to a finite energy budget: cells highly proficient in chemotaxis exhibit lower proliferation rates, while more proliferative cells show diminished chemotactic abilities. The model is derived from the principles of a biased random walk, resulting in a system of two non-local partial differential equations, describing the densities of both cells and ligands. Using a Hopf-Cole transformation, we derive an equation that characterises the distribution of cellular traits within travelling wave solutions for the total cell density, allowing us to uncover the monotonicity properties of these waves. Numerical investigations are conducted to examine the model's behaviour across various biological scenarios, providing insights into the complex interplay between chemotaxis, proliferation, and phenotypic diversity in migrating cell populations.
Genotype-to-Phenotype Prediction in Rice with High-Dimensional Nonlinear Features
Authors: Zeyuan Zhou, Siyuan Chen, Xinzhang Wu, Jisen Zhang, Yunxuan Dong
Date: 2025-02-26 00:00:00
Tasks: Prediction

Genotype-to-Phenotype prediction can promote advances in modern genomic research and crop improvement, guiding precision breeding and genomic selection. However, high-dimensional nonlinear features often hinder the accuracy of genotype-to-phenotype prediction by increasing computational complexity. The challenge also limits the predictive accuracy of traditional approaches. Therefore, effective solutions are needed to improve the accuracy of genotype-to-phenotype prediction. In our paper, we propose MLFformer. MLFformer is a Transformer-based architecture that incorporates the Fast Attention mechanism and a multilayer perceptron module to handle high-dimensional nonlinear features. In MLFformer, the Fast Attention mechanism is utilized to handle computational complexity and enhance processing efficiency. In addition, the MLP structure further captures high-dimensional nonlinear features. Through experiments, the results show that MLFformer reduces the average MAPE by 7.73% compared to the vanilla Transformer. In univariate and multivariate prediction scenarios, MLFformer achieves the best predictive performance among all compared models.
scMamba: A Pre-Trained Model for Single-Nucleus RNA Sequencing Analysis in Neurodegenerative Disorders
Authors: Gyutaek Oh, Baekgyu Choi, Seyoung Jin, Inkyung Jung, Jong Chul Ye
Date: 2025-02-12 00:00:00
Tasks: Dimensionality Reduction, Imputation, Mamba

Single-nucleus RNA sequencing (snRNA-seq) has significantly advanced our understanding of the disease etiology of neurodegenerative disorders. However, the low quality of specimens derived from postmortem brain tissues, combined with the high variability caused by disease heterogeneity, makes it challenging to integrate snRNA-seq data from multiple sources for precise analyses. To address these challenges, we present scMamba, a pre-trained model designed to improve the quality and utility of snRNA-seq analysis, with a particular focus on neurodegenerative diseases. Inspired by the recent Mamba model, scMamba introduces a novel architecture that incorporates a linear adapter layer, gene embeddings, and bidirectional Mamba blocks, enabling efficient processing of snRNA-seq data while preserving information from the raw input. Notably, scMamba learns generalizable features of cells and genes through pre-training on snRNA-seq data, without relying on dimension reduction or selection of highly variable genes. We demonstrate that scMamba outperforms benchmark methods in various downstream tasks, including cell type annotation, doublet detection, imputation, and the identification of differentially expressed genes.
Modelling Chemical Reaction Networks using Neural Ordinary Differential Equations
Authors: Anna C. M. Thöni, William E. Robinson, Yoram Bachrach, Wilhelm T. S. Huck, Tal Kachman
Date: 2025-02-11 00:00:00
Tasks: Form

In chemical reaction network theory, ordinary differential equations are used to model the temporal change of chemical species concentration. As the functional form of these ordinary differential equations systems is derived from an empirical model of the reaction network, it may be incomplete. Our approach aims to elucidate these hidden insights in the reaction network by combining dynamic modelling with deep learning in the form of neural ordinary differential equations. Our contributions not only help to identify the shortcomings of existing empirical models but also assist the design of future reaction networks.
Naturalistic Computational Cognitive Science: Towards generalizable models and theories that capture the full range of natural behavior
Authors: Wilka Carvalho, Andrew Lampinen
Date: 2025-02-27 00:00:00

How can cognitive science build generalizable theories that span the full scope of natural situations and behaviors? We argue that progress in Artificial Intelligence (AI) offers timely opportunities for cognitive science to embrace experiments with increasingly naturalistic stimuli, tasks, and behaviors; and computational models that can accommodate these changes. We first review a growing body of research spanning neuroscience, cognitive science, and AI that suggests that incorporating a broader range of naturalistic experimental paradigms, and models that accommodate them, may be necessary to resolve some aspects of natural intelligence and ensure that our theories generalize. First, we review cases from cognitive science and neuroscience where naturalistic paradigms elicit distinct behaviors or engage different processes. We then discuss recent progress in AI that shows that learning from naturalistic data yields qualitatively different patterns of behavior and generalization, and discuss how these findings impact the conclusions we draw from cognitive modeling, and can help yield new hypotheses for the roots of cognitive and neural phenomena. We then suggest that integrating recent progress in AI and cognitive science will enable us to engage with more naturalistic phenomena without giving up experimental control or the pursuit of theoretically grounded understanding. We offer practical guidance on how methodological practices can contribute to cumulative progress in naturalistic computational cognitive science, and illustrate a path towards building computational models that solve the real problems of natural cognition, together with a reductive understanding of the processes and principles by which they do so.
NeuroTree: Hierarchical Functional Brain Pathway Decoding for Mental Health Disorders
Authors: Jun-En Ding, Dongsheng Luo, Anna Zilverstand, Kaustubh Kulkarni, Feng Liu
Date: 2025-02-26 00:00:00
Tasks: Functional Connectivity

Mental disorders are among the most widespread diseases globally. Analyzing functional brain networks through functional magnetic resonance imaging (fMRI) is crucial for understanding mental disorder behaviors. Although existing fMRI-based graph neural networks (GNNs) have demonstrated significant potential in brain network feature extraction, they often fail to characterize complex relationships between brain regions and demographic information in mental disorders. To overcome these limitations, we propose a learnable NeuroTree framework that integrates a k-hop AGE-GCN with neural ordinary differential equations (ODEs) and contrastive masked functional connectivity (CMFC) to enhance similarities and dissimilarities of brain region distance. Furthermore, NeuroTree effectively decodes fMRI network features into tree structures, which improves the capture of high-order brain regional pathway features and enables the identification of hierarchical neural behavioral patterns essential for understanding disease-related brain subnetworks. Our empirical evaluations demonstrate that NeuroTree achieves state-of-the-art performance across two distinct mental disorder datasets. It provides valuable insights into age-related deterioration patterns, elucidating their underlying neural mechanisms.