pwc-archive/papers-with-abstracts huggingface dataset:
Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension
Authors: Moritz Haas, David Holzmüller, Ulrike Von Luxburg, Ingo Steinwart
Date: 2023-05-23 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: regression

The success of over-parameterized neural networks trained to near-zero training error has caused great interest in the phenomenon of benign overfitting, where estimators are statistically consistent even though they interpolate noisy training data. While benign overfitting in fixed dimension has been established for some learning methods, current literature suggests that for regression with typical kernel methods and wide neural networks, benign overfitting requires a high-dimensional setting where the dimension grows with the sample size. In this paper, we show that the smoothness of the estimators, and not the dimension, is the key: benign overfitting is possible if and only if the estimator's derivatives are large enough. We generalize existing inconsistency results to non-interpolating models and more kernels to show that benign overfitting with moderate derivatives is impossible in fixed dimension. Conversely, we show that rate-optimal benign overfitting is possible for regression with a sequence of spiky-smooth kernels with large derivatives. Using neural tangent kernels, we translate our results to wide neural networks. We prove that while infinite-width networks do not overfit benignly with the ReLU activation, this can be fixed by adding small high-frequency fluctuations to the activation function. Our experiments verify that such neural networks, while overfitting, can indeed generalize well even on low-dimensional data sets.
A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods
Authors: Veit David Wild, Sahra Ghalebikesabi, Dino Sejdinovic, Jeremias Knoblauch
Date: 2023-05-24 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Deep Learning, Uncertainty Quantification, Variational Inference

We establish the first mathematically rigorous link between Bayesian, variational Bayesian, and ensemble methods. A key step towards this it to reformulate the non-convex optimisation problem typically encountered in deep learning as a convex optimisation in the space of probability measures. On a technical level, our contribution amounts to studying generalised variational inference through the lense of Wasserstein gradient flows. The result is a unified theory of various seemingly disconnected approaches that are commonly used for uncertainty quantification in deep learning -- including deep ensembles and (variational) Bayesian methods. This offers a fresh perspective on the reasons behind the success of deep ensembles over procedures based on parameterised variational inference, and allows the derivation of new ensembling schemes with convergence guarantees. We showcase this by proposing a family of interacting deep ensembles with direct parallels to the interactions of particle systems in thermodynamics, and use our theory to prove the convergence of these algorithms to a well-defined global minimiser on the space of probability measures.
Towards a Unified Analysis of Kernel-based Methods Under Covariate Shift
Authors: Xingdong Feng, Xin He, Caixing Wang, Chao Wang, Jingnan Zhang
Date: 2023-10-12 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: quantile regression, regression

Covariate shift occurs prevalently in practice, where the input distributions of the source and target data are substantially different. Despite its practical importance in various learning problems, most of the existing methods only focus on some specific learning tasks and are not well validated theoretically and numerically. To tackle this problem, we propose a unified analysis of general nonparametric methods in a reproducing kernel Hilbert space (RKHS) under covariate shift. Our theoretical results are established for a general loss belonging to a rich loss function family, which includes many commonly used methods as special cases, such as mean regression, quantile regression, likelihood-based classification, and margin-based classification. Two types of covariate shift problems are the focus of this paper and the sharp convergence rates are established for a general loss function to provide a unified theoretical analysis, which concurs with the optimal results in literature where the squared loss is used. Extensive numerical studies on synthetic and real examples confirm our theoretical findings and further illustrate the effectiveness of our proposed method.
NMR Spectroscopy Can Help Accelerate Antiviral Drug Discovery Programs
Authors: Steven R. Laplante, Pascale Coric, Serge Bouaziz, Tanos Celmar Costa França
Date: 2023-11-29 00:00:00
Tasks: Computational chemistry, Drug Discovery

Small molecule drugs have an important role to play in combating viral infections, and biophysics support has been central for contributing to the discovery and design of direct acting antivirals. Perhaps one of the most successful biophysical tools for this purpose is NMR spectroscopy when utilized strategically and pragmatically within team workflows and timelines. This report describes some clear examples of how NMR applications contributed to the design of antivirals when combined with medicinal chemistry, biochemistry, X-ray crystallography and computational chemistry. Overall, these multidisciplinary approaches allowed teams to reveal and expose compound physical properties from which design ideas were spawned and tested to achieve the desired successes. Examples are discussed for the discovery of antivirals that target HCV, HIV and SARS-CoV-2.
Experimental and Theoretical Brownian Dynamics Analysis of Ion Transport During Cellular Electroporation of E. coli Bacteria
Authors: Juan González-Cuevas, Ricardo Argüello, Marcos Florentin, Franck M. André, Lluis Mir
Date: 2023-11-29 00:00:00

Escherichia coli bacterium is a rod-shaped organism composed of a complex double membrane structure. Knowledge of electric field driven ion transport through both membranes and the evolution of their induced permeabilization has important applications in biomedical engineering, delivery of genes and antibacterial agents. However, few studies have been conducted on Gram-negative bacteria in this regard considering the contribution of all ion types. To address this gap in knowledge, we have developed a deterministic and stochastic Brownian dynamics model to simulate in 3D space the motion of ions through pores formed in the plasma membranes of E. coli cells during electroporation. The diffusion coefficient, mobility, and translation time of Ca$^{2+}$, Mg$^{2+}$, Na$^+$, K$^+$, and Cl$^-$ ions within the pore region are estimated from the numerical model. Calculations of pore's conductance have been validated with experiments conducted at Gustave Roussy. From the simulations, it was found that the main driving force of ionic uptake during the pulse is the one due to the externally applied electric field. The results from this work provide a better understanding of ion transport during electroporation, aiding in the design of electrical pulses for maximizing ion throughput, primarily for application in cancer treatment.
Community recommendations on cryoEM data archiving and validation
Authors: Gerard J. Kleywegt, Paul D. Adams, Sarah J. Butcher, Cathy Lawson, Alexis Rohou, Peter B. Rosenthal, Sriram Subramaniam, Maya Topf, Sanja Abbott, Philip R. Baldwin, John M. Berrisford, Gérard Bricogne, Preeti Choudhary, Tristan I. Croll, Radostin Danev, Sai J. Ganesan, Timothy Grant, Aleksandras Gutmanas, Richard Henderson, J. Bernard Heymann, Juha T. Huiskonen, Andrei Istrate, Takayuki Kato, Gabriel C. Lander, Shee-Mei Lok, Steven J. Ludtke, Garib N. Murshudov, Ryan Pye, Grigore D. Pintilie, Jane S. Richardson, Carsten Sachse, Osman Salih, Sjors H. W. Scheres, Gunnar F. Schroeder, Carlos Oscar S. Sorzano, Scott M. Stagg, Zhe Wang, Rangana Warshamanage, John D. Westbrook, Martyn D. Winn, Jasmine Y. Young, Stephen K. Burley, Jeffrey C. Hoch, Genji Kurisu, Kyle Morris, Ardan Patwardhan, Sameer Velankar
Date: 2023-11-29 00:00:00
Tasks: Single Particle Analysis

In January 2020, a workshop was held at EMBL-EBI (Hinxton, UK) to discuss data requirements for deposition and validation of cryoEM structures, with a focus on single-particle analysis. The meeting was attended by 45 experts in data processing, model building and refinement, validation, and archiving of such structures. This report describes the workshop's motivation and history, the topics discussed, and consensus recommendations resulting from the workshop. Some challenges for future methods-development efforts in this area are also highlighted, as is the implementation to date of some of the recommendations.
Identification of urinary biomarkers of food intake for onion by untargeted LC-MS metabolomics
Authors: Qian Gao, G. Gurdeniz, Giulia Pratico, Camilla T. Damsgaard, Eduvigis Roldan-Marin, M. Pilar Cano, Concepcion Sanchez-Moreno, Lars O. Dragsted
Date: 2023-11-28 00:00:00

Scope: Biomarkers of food intake (BFIs) are useful tools for objective assessment of food intake and compliance. The aim of this study was to discover and identify urinary BFIs for onion. Methods and results: In a randomized controlled cross-over trial, 6 overweight participants (age 24-62 years) consumed meals with 20 g/d onion powder or no onion for 2 weeks. Untargeted UPLC-qTOF-MS metabolic profiling analysis was performed on urine samples and the profiles were analysed by multilevel-PLSDA, modified PLS, and nearest shrunken centroid to select features associated with onion intake. Eight biomarkers were tentatively identified; six of them originated from S-substituted cysteine derivatives such as isoalliin and propiin, which are considered the most specific for onion intake. Most of the biomarkers were completely excreted within 24 hours and no accumulation was observed during 2 weeks indicating their ability to reflect only recent intake of onions. Receiver-operator curves were made to evaluate the performance of individual biomarkers for predicting onion intake. The area under the curve values for these biomarkers ranged from 0.81 to 1. Conclusion: Promising biomarkers of recent onion intake have been identified in human urine. Further studies with complex diets are needed to validate the robustness of these biomarkers.
Predicting a Protein's Stability under a Million Mutations
Authors: Jeffrey Ouyang-Zhang, Daniel J. Diaz, Adam R. Klivans, Philipp Krähenbühl
Date: 2023-10-19 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Protein Stability Prediction

Stabilizing proteins is a foundational step in protein engineering. However, the evolutionary pressure of all extant proteins makes identifying the scarce number of mutations that will improve thermodynamic stability challenging. Deep learning has recently emerged as a powerful tool for identifying promising mutations. Existing approaches, however, are computationally expensive, as the number of model inferences scales with the number of mutations queried. Our main contribution is a simple, parallel decoding algorithm. Our Mutate Everything is capable of predicting the effect of all single and double mutations in one forward pass. It is even versatile enough to predict higher-order mutations with minimal computational overhead. We build Mutate Everything on top of ESM2 and AlphaFold, neither of which were trained to predict thermodynamic stability. We trained on the Mega-Scale cDNA proteolysis dataset and achieved state-of-the-art performance on single and higher-order mutations on S669, ProTherm, and ProteinGym datasets. Code is available at https://github.com/jozhang97/MutateEverything
GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning
Authors: Haiteng Zhao, Shengchao Liu, Chang Ma, Hannan Xu, Jie Fu, Zhi-Hong Deng, Lingpeng Kong, Qi Liu
Date: 2023-05-28 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Property Prediction, Zero-Shot Learning

Molecule property prediction has gained significant attention in recent years. The main bottleneck is the label insufficiency caused by expensive lab experiments. In order to alleviate this issue and to better leverage textual knowledge for tasks, this study investigates the feasibility of employing natural language instructions to accomplish molecule-related tasks in a zero-shot setting. We discover that existing molecule-text models perform poorly in this setting due to inadequate treatment of instructions and limited capacity for graphs. To overcome these issues, we propose GIMLET, which unifies language models for both graph and text data. By adopting generalized position embedding, our model is extended to encode both graph structures and instruction text without additional graph encoding modules. GIMLET also decouples encoding of the graph from tasks instructions in the attention mechanism, enhancing the generalization of graph features across novel tasks. We construct a dataset consisting of more than two thousand molecule tasks with corresponding instructions derived from task descriptions. We pretrain GIMLET on the molecule tasks along with instructions, enabling the model to transfer effectively to a broad range of tasks. Experimental results demonstrate that GIMLET significantly outperforms molecule-text baselines in instruction-based zero-shot learning, even achieving closed results to supervised GNN models on tasks such as toxcast and muv.
Full-Atom Protein Pocket Design via Iterative Refinement
Authors: Zaixi Zhang, Zepu Lu, Zhongkai Hao, Marinka Zitnik, Qi Liu
Date: 2023-10-04 00:00:00
Proceeding: NeurIPS 2023 11

The design of \emph{de novo} functional proteins that bind specific ligand molecules is paramount in therapeutics and bio-engineering. A critical yet formidable task in this endeavor is the design of the protein pocket, which is the cavity region of the protein where the ligand binds. Current methods are plagued by inefficient generation, inadequate context modeling of the ligand molecule, and the inability to generate side-chain atoms. Here, we present the Full-Atom Iterative Refinement (FAIR) method, designed to address these challenges by facilitating the co-design of protein pocket sequences, specifically residue types, and their corresponding 3D structures. FAIR operates in two steps, proceeding in a coarse-to-fine manner (transitioning from protein backbone to atoms, including side chains) for a full-atom generation. In each iteration, all residue types and structures are simultaneously updated, a process termed full-shot refinement. In the initial stage, the residue types and backbone coordinates are refined using a hierarchical context encoder, complemented by two structure refinement modules that capture both inter-residue and pocket-ligand interactions. The subsequent stage delves deeper, modeling the side-chain atoms of the pockets and updating residue types to ensure sequence-structure congruence. Concurrently, the structure of the binding ligand is refined across iterations to accommodate its inherent flexibility. Comprehensive experiments show that FAIR surpasses existing methods in designing superior pocket sequences and structures, producing average improvement exceeding 10\% in AAR and RMSD metrics. FAIR is available at \url{https://github.com/zaixizhang/FAIR}.
Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

Vision-language pre-training methods, e.g., CLIP, demonstrate an impressive zero-shot performance on visual categorizations with the class proxy from the text embedding of the class name. However, the modality gap between the text and vision space can result in a sub-optimal performance. We theoretically show that the gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP and the optimal proxy for vision tasks may reside only in the vision space. Therefore, given unlabeled target vision data, we propose to learn the vision proxy directly with the help from the text proxy for zero-shot transfer. Moreover, according to our theoretical analysis, strategies are developed to further refine the pseudo label obtained by the text proxy to facilitate the intra-modal proxy learning (InMaP) for vision. Experiments on extensive downstream tasks confirm the effectiveness and efficiency of our proposal. Concretely, InMaP can obtain the vision proxy within one minute on a single GPU while improving the zero-shot accuracy from $77.02\%$ to $80.21\%$ on ImageNet with ViT-L/14@336 pre-trained by CLIP.Submission Number: 10968
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Authors: Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, Chris Ré
Date: 2023-06-27 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: 4k, In-Context Learning, Language Modelling, Large Language Model

Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at https://github.com/HazyResearch/hyena-dna.
The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance
Authors: Jon Donnelly, Srikar Katta, Cynthia Rudin, Edward P. Browne
Date: 2023-09-24 00:00:00
Proceeding: NeurIPS 2023 11

Quantifying variable importance is essential for answering high-stakes questions in fields like genetics, public policy, and medicine. Current methods generally calculate variable importance for a given model trained on a given dataset. However, for a given dataset, there may be many models that explain the target outcome equally well; without accounting for all possible explanations, different researchers may arrive at many conflicting yet equally valid conclusions given the same data. Additionally, even when accounting for all possible explanations for a given dataset, these insights may not generalize because not all good explanations are stable across reasonable data perturbations. We propose a new variable importance framework that quantifies the importance of a variable across the set of all good models and is stable across the data distribution. Our framework is extremely flexible and can be integrated with most existing model classes and global variable importance metrics. We demonstrate through experiments that our framework recovers variable importance rankings for complex simulation setups where other methods fail. Further, we show that our framework accurately estimates the true importance of a variable for the underlying data distribution. We provide theoretical guarantees on the consistency and finite sample error rates for our estimator. Finally, we demonstrate its utility with a real-world case study exploring which genes are important for predicting HIV load in persons with HIV, highlighting an important gene that has not previously been studied in connection with HIV. Code is available at https://github.com/jdonnelly36/Rashomon_Importance_Distribution.
An Algorithm Based on a Cable-Nernst Planck Model Predicting Synaptic Activity throughout the Dendritic Arbor with Micron Specificity
Authors: Claire Guerrier, Tristan Dellazizzo Toth, Nicolas Galtier, Kurt Haas
Date: 2023-11-29 00:00:00
Tasks: Specificity

Recent technological advances have enabled the recording of neurons in intact circuits with a high spatial and temporal resolution, creating the need for modeling with the same precision. In particular, the development of ultra-fast two-photon microscopy combined with fluorescence-based genetically-encoded Ca2+-indicators allows capture of full-dendritic arbor and somatic responses associated with synaptic input and action potential output. The complexity of dendritic arbor structures and distributed patterns of activity over time results in the generation of incredibly rich 4D datasets that are challenging to analyze (Sakaki, 2020). Interpreting neural activity from fluorescence-based Ca2+ biosensors is challenging due to non-linear interactions between several factors influencing intracellular calcium ion concentration and its binding to sensors, including the ionic dynamics driven by diffusion, electrical gradients and voltage-gated conductance.To investigate those dynamics, we designed a model based on a Cable-like equation coupled to the Nernst-Planck equations for ionic fluxes in electrolytes. We employ this model to simulate signal propagation and ionic electrodiffusion across a dendritic arbor. Using these simulation results, we then designed an algorithm to detect synapses from Ca2+ imaging datasets. We finally apply this algorithm to experimental Ca2+-indicator datasets from neurons expressing jGCaMP7s (Dana et al., 2019), using full-dendritic arbor sampling in vivo in the Xenopus laevis optic tectum using fast random-access two-photon microscopy.Our model reproduces the dynamics of visual stimulus-evoked jGCaMP7s-mediated calcium signals observed experimentally, and the resulting algorithm allows prediction of the location of synapses across the dendritic arbor.Our study provides a way to predict synaptic activity and location on dendritic arbors, from fluorescence data in the full dendritic arbor of a neuron recorded in the intact and awake developing vertebrate brain.