pwc-archive/papers-with-abstracts huggingface dataset:
A SSM is Polymerized from Multivariate Time Series
Authors: Haixiang Wu
Date: 2024-09-30 00:00:00
Tasks: Mamba, State Space Models, Time Series

For multivariate time series (MTS) tasks, previous state space models (SSMs) followed the modeling paradigm of Transformer-based methods. However, none of them explicitly model the complex dependencies of MTS: the Channel Dependency variations with Time (CDT). In view of this, we delve into the derivation of SSM, which involves approximating continuously updated functions by orthogonal function basis. We then develop Poly-Mamba, a novel method for MTS forecasting. Its core concept is to expand the original orthogonal function basis space into a multivariate orthogonal function space containing variable mixing terms, and make a projection on this space so as to explicitly describe the CDT by weighted coefficients. In Poly-Mamba, we propose the Multivariate Orthogonal Polynomial Approximation (MOPA) as a simplified implementation of this concept. For the simple linear relationship between channels, we propose Linear Channel Mixing (LCM) and generate CDT patterns adaptively for different channels through a proposed Order Combining method. Experiments on six real-world datasets demonstrate that Poly-Mamba outperforms the SOTA methods, especially when dealing with datasets having a large number of channels and complex correlations. The codes and log files will be released at: https://github.com/Joeland4/Poly-Mamba.
PersonalLLM: Tailoring LLMs to Individual Preferences
Authors: Thomas P. Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, Hongseok Namkoong
Date: 2024-09-30 00:00:00
Tasks: In-Context Learning, Meta-Learning

As LLMs become capable of complex tasks, there is growing potential for personalized interactions tailored to the subtle and idiosyncratic preferences of the user. We present a public benchmark, PersonalLLM, focusing on adapting LLMs to provide maximal benefits for a particular user. Departing from existing alignment benchmarks that implicitly assume uniform preferences, we curate open-ended prompts paired with many high-quality answers over which users would be expected to display heterogeneous latent preferences. Instead of persona-prompting LLMs based on high-level attributes (e.g., user's race or response length), which yields homogeneous preferences relative to humans, we develop a method that can simulate a large user base with diverse preferences from a set of pre-trained reward models. Our dataset and generated personalities offer an innovative testbed for developing personalization algorithms that grapple with continual data sparsity--few relevant feedback from the particular user--by leveraging historical data from other (similar) users. We explore basic in-context learning and meta-learning baselines to illustrate the utility of PersonalLLM and highlight the need for future methodological development. Our dataset is available at https://huggingface.co/datasets/namkoong-lab/PersonalLLM
First Order System Least Squares Neural Networks
Authors: Joost A. A. Opschoor, Philipp C. Petersen, Christoph Schwab
Date: 2024-09-30 00:00:00

We introduce a conceptual framework for numerically solving linear elliptic, parabolic, and hyperbolic PDEs on bounded, polytopal domains in euclidean spaces by deep neural networks. The PDEs are recast as minimization of a least-squares (LSQ for short) residual of an equivalent, well-posed first-order system, over parametric families of deep neural networks. The associated LSQ residual is a) equal or proportional to a weak residual of the PDE, b) additive in terms of contributions from localized subnetworks, indicating locally ``out-of-equilibrium'' of neural networks with respect to the PDE residual, c) serves as numerical loss function for neural network training, and d) constitutes, even with incomplete training, a computable, (quasi-)optimal numerical error estimator in the context of adaptive LSQ finite element methods. In addition, an adaptive neural network growth strategy is proposed which, assuming exact numerical minimization of the LSQ loss functional, yields sequences of neural networks with realizations that converge rate-optimally to the exact solution of the first order system LSQ formulation.
Controlling sharpness, SNR and SAR for 3D FSE at 7T by end-to-end learning
Authors: Peter Dawood, Martin Blaimer, Jürgen Herrler, Patrick Liebig, Simon Weinmüller, Shaihan Malik, Peter M. Jakob, Moritz Zaiss
Date: 2024-09-30 00:00:00

Purpose: To non-heuristically identify dedicated variable flip angle (VFA) schemes optimized for the point-spread function (PSF) and signal-to-noise ratio (SNR) of multiple tissues in 3D FSE sequences with very long echo trains at 7T. Methods: The proposed optimization considers predefined SAR constraints and target contrast using an end-to-end learning framework. The cost function integrates components for contrast fidelity (SNR) and a penalty term to minimize image blurring (PSF) for multiple tissues. By adjusting the weights of PSF/SNR cost-function components, PSF- and SNR-optimized VFAs were derived and tested in vivo using both the open-source Pulseq standard on two volunteers as well as vendor protocols on a 7T MRI system with parallel transmit extension on three volunteers. Results: PSF-optimized VFAs resulted in significantly reduced image blurring compared to standard VFAs for T2w while maintaining contrast fidelity. Small white and gray matter structures, as well as blood vessels, are more visible with PSF-optimized VFAs. Quantitative analysis shows that the optimized VFA yields 50% less deviation from a sinc-like reference PSF than the standard VFA. The SNR-optimized VFAs yielded images with significantly improved SNR in a white and gray matter region relative to standard (81.2\pm18.4 vs. 41.2\pm11.5, respectively) as trade-off for elevated image blurring. Conclusion: This study demonstrates the potential of end-to-end learning frameworks to optimize VFA schemes in very long echo trains for 3D FSE acquisition at 7T in terms of PSF and SNR. It paves the way for fast and flexible adjustment of the trade-off between PSF and SNR for 3D FSE.
Random Features Outperform Linear Models: Effect of Strong Input-Label Correlation in Spiked Covariance Data
Authors: Samet Demir, Zafer Dogan
Date: 2024-09-30 00:00:00

Random Feature Model (RFM) with a nonlinear activation function is instrumental in understanding training and generalization performance in high-dimensional learning. While existing research has established an asymptotic equivalence in performance between the RFM and noisy linear models under isotropic data assumptions, empirical observations indicate that the RFM frequently surpasses linear models in practical applications. To address this gap, we ask, "When and how does the RFM outperform linear models?" In practice, inputs often have additional structures that significantly influence learning. Therefore, we explore the RFM under anisotropic input data characterized by spiked covariance in the proportional asymptotic limit, where dimensions diverge jointly while maintaining finite ratios. Our analysis reveals that a high correlation between inputs and labels is a critical factor enabling the RFM to outperform linear models. Moreover, we show that the RFM performs equivalent to noisy polynomial models, where the polynomial degree depends on the strength of the correlation between inputs and labels. Our numerical simulations validate these theoretical insights, confirming the performance-wise superiority of RFM in scenarios characterized by strong input-label correlation.
A general machine learning model of aluminosilicate melt viscosity and its application to the surface properties of dry lava planets
Authors: Charles Le Losq, Clément Ferraina, Paolo A. Sossi, Charles-Édouard Boukaré
Date: 2024-09-30 00:00:00

Ultra-short-period exoplanets like K2-141 b likely have magma oceans on their dayside, which play a critical role in redistributing heat within the planet. This could lead to a warm nightside surface, measurable by the James Webb Space Telescope, offering insights into the planet's structure. Accurate models of properties like viscosity, which can vary by orders of magnitude, are essential for such studies. We present a new model for predicting molten magma viscosity, applicable in diverse scenarios, including magma oceans on lava planets. Using a database of 28,898 viscosity measurements on phospho-alumino-silicate melts, spanning superliquidus to undercooled temperatures and pressures up to 30 GPa, we trained a greybox artificial neural network, refined by a Gaussian process. This model achieves high predictive accuracy (RMSE $\approx 0.4 \log_{10}$ Pa$\cdot$s) and can handle compositions from SiO$_2$ to multicomponent magmatic and industrial glasses, accounting for pressure effects up to 30 GPa for compositions such as peridotite. Applying this model, we calculated the viscosity of K2-141 b's magma ocean under different compositions. Phase diagram calculations suggest that the dayside is fully molten, with extreme temperatures primarily controlling viscosity. A tenuous atmosphere (0.1 bar) might exist around a 40{\deg} radius from the substellar point. At higher longitudes, atmospheric pressure drops, and by 90{\deg}, magma viscosity rapidly increases as solidification occurs. The nightside surface is likely solid, but previously estimated surface temperatures above 400 K imply a partly molten mantle, feeding geothermal flux through vertical convection.
Assessing interaction recovery of predicted protein-ligand poses
Authors: David Errington, Constantin Schneider, Cédric Bouysset, Frédéric A. Dreyer
Date: 2024-09-30 00:00:00
Tasks: Pose Prediction

The field of protein-ligand pose prediction has seen significant advances in recent years, with machine learning-based methods now being commonly used in lieu of classical docking methods or even to predict all-atom protein-ligand complex structures. Most contemporary studies focus on the accuracy and physical plausibility of ligand placement to determine pose quality, often neglecting a direct assessment of the interactions observed with the protein. In this work, we demonstrate that ignoring protein-ligand interaction fingerprints can lead to overestimation of model performance, most notably in recent protein-ligand cofolding models which often fail to recapitulate key interactions.
Constraining Anomaly Detection with Anomaly-Free Regions
Authors: Maximilian Toller, Hussain Hussain, Roman Kern, Bernhard C. Geiger
Date: 2024-09-30 00:00:00
Tasks: Anomaly Detection

We propose the novel concept of anomaly-free regions (AFR) to improve anomaly detection. An AFR is a region in the data space for which it is known that there are no anomalies inside it, e.g., via domain knowledge. This region can contain any number of normal data points and can be anywhere in the data space. AFRs have the key advantage that they constrain the estimation of the distribution of non-anomalies: The estimated probability mass inside the AFR must be consistent with the number of normal data points inside the AFR. Based on this insight, we provide a solid theoretical foundation and a reference implementation of anomaly detection using AFRs. Our empirical results confirm that anomaly detection constrained via AFRs improves upon unconstrained anomaly detection. Specifically, we show that, when equipped with an estimated AFR, an efficient algorithm based on random guessing becomes a strong baseline that several widely-used methods struggle to overcome. On a dataset with a ground-truth AFR available, the current state of the art is outperformed.
SetPINNs: Set-based Physics-informed Neural Networks
Authors: Mayank Nagda, Phil Ostheimer, Thomas Specht, Frank Rhein, Fabian Jirasek, Stephan Mandt, Marius Kloft, Sophie Fellenz
Date: 2024-09-30 00:00:00

Physics-Informed Neural Networks (PINNs) solve partial differential equations using deep learning. However, conventional PINNs perform pointwise predictions that neglect dependencies within a domain, which may result in suboptimal solutions. We introduce SetPINNs, a framework that effectively captures local dependencies. With a finite element-inspired sampling scheme, we partition the domain into sets to model local dependencies while simultaneously enforcing physical laws. We provide a rigorous theoretical analysis showing that SetPINNs yield unbiased, lower-variance estimates of residual energy and its gradients, ensuring improved domain coverage and reduced residual error. Extensive experiments on synthetic and real-world tasks show improved accuracy, efficiency, and robustness.
Ensemble Kalman Diffusion Guidance: A Derivative-free Method for Inverse Problems
Authors: Hongkai Zheng, Wenda Chu, Austin Wang, Nikola Kovachki, Ricardo Baptista, Yisong Yue
Date: 2024-09-30 00:00:00

When solving inverse problems, it is increasingly popular to use pre-trained diffusion models as plug-and-play priors. This framework can accommodate different forward models without re-training while preserving the generative capability of diffusion models. Despite their success in many imaging inverse problems, most existing methods rely on privileged information such as derivative, pseudo-inverse, or full knowledge about the forward model. This reliance poses a substantial limitation that restricts their use in a wide range of problems where such information is unavailable, such as in many scientific applications. To address this issue, we propose Ensemble Kalman Diffusion Guidance (EnKG) for diffusion models, a derivative-free approach that can solve inverse problems by only accessing forward model evaluations and a pre-trained diffusion model prior. We study the empirical effectiveness of our method across various inverse problems, including scientific settings such as inferring fluid flows and astronomical objects, which are highly non-linear inverse problems that often only permit black-box access to the forward model.
ASTRA: Accurate and Scalable ANNS-based Training of Extreme Classifiers
Authors: Sonu Mehta, Jayashree Mohan, Nagarajan Natarajan, Ramachandran Ramjee, Manik Varma
Date: 2024-09-30 00:00:00

`Extreme Classification'' (or XC) is the task of annotating data points (queries) with relevant labels (documents), from an extremely large set of $L$ possible labels, arising in search and recommendations. The most successful deep learning paradigm that has emerged over the last decade or so for XC is to embed the queries (and labels) using a deep encoder (e.g. DistilBERT), and use linear classifiers on top of the query embeddings. This architecture is of appeal because it enables millisecond-time inference using approximate nearest neighbor search (ANNS). The key question is how do we design training algorithms that are accurate as well as scale to $O(100M)$ labels on a limited number of GPUs. State-of-the-art XC techniques that demonstrate high accuracies (e.g., DEXML, Ren\'ee, DEXA) on standard datasets have per-epoch training time that scales as $O(L)$ or employ expensive negative sampling strategies, which are prohibitive in XC scenarios. In this work, we develop an accurate and scalable XC algorithm ASTRA with two key observations: (a) building ANNS index on the classifier vectors and retrieving hard negatives using the classifiers aligns the negative sampling strategy to the loss function optimized; (b) keeping the ANNS indices current as the classifiers change through the epochs is prohibitively expensive while using stale negatives (refreshed periodically) results in poor accuracy; to remedy this, we propose a negative sampling strategy that uses a mixture of importance sampling and uniform sampling. By extensive evaluation on standard XC as well as proprietary datasets with 120M labels, we demonstrate that ASTRA achieves SOTA precision, while reducing training time by 4x-15x relative to the second best.
Constraint Guided Model Quantization of Neural Networks
Authors: Quinten Van Baelen, Peter Karsmakers
Date: 2024-09-30 00:00:00
Tasks: model, Quantization

Deploying neural networks on the edge has become increasingly important as deep learning is being applied in an increasing amount of applications. The devices on the edge are typically characterised as having small computational resources as large computational resources results in a higher energy consumption, which is impractical for these devices. To reduce the complexity of neural networks a wide range of quantization methods have been proposed in recent years. This work proposes Constraint Guided Model Quantization (CGMQ), which is a quantization aware training algorithm that uses an upper bound on the computational resources and reduces the bit-widths of the parameters of the neural network. CGMQ does not require the tuning of a hyperparameter to result in a mixed precision neural network that satisfies the predefined computational cost constraint, while prior work does. It is shown on MNIST that the performance of CGMQ is competitive with state-of-the-art quantization aware training algorithms, while guaranteeing the satisfaction of the cost constraint.
Federated Instruction Tuning of LLMs with Domain Coverage Augmentation
Authors: Zezhou Wang, Yaxin Du, Xingjun Ma, Yugang Jiang, Zhuzhong Qian, Siheng Chen
Date: 2024-09-30 00:00:00
Tasks: Computational Efficiency, Privacy Preserving

Federated Domain-specific Instruction Tuning (FedDIT) utilizes limited cross-client private data together with various strategies of instruction augmentation, ultimately boosting model performance within specific domains. To date, the factors affecting FedDIT remain unclear, and existing instruction augmentation methods primarily focus on the centralized setting without considering distributed environments. Our experiments reveal that the cross-client domain coverage, rather than data heterogeneity, drives model performance in FedDIT. In response, we propose FedDCA, which optimizes domain coverage through greedy client center selection and retrieval-based augmentation. At its core, the greedy selection procedure iteratively picks client centers that maximize the diversity and coverage of the instruction space while avoiding redundancy with previously selected centers. This ensures broad yet efficient coverage of the domain distribution across clients. For client-side computational efficiency and system scalability, FedDCA$^*$, the variant of FedDCA, utilizes heterogeneous encoders with server-side feature alignment. Extensive experiments across code, medical, financial, and mathematical domains substantiate the effectiveness of both methods, as well as plug-and-play capability. We further analyze privacy preservation against memory extraction attacks, showing that while privacy leakage risk is independent of augmented public data ratio, it decreases or converges as training progresses.
DCAST: Diverse Class-Aware Self-Training Mitigates Selection Bias for Fairer Learning
Authors: Yasin I. Tepeli, Joana P. Gonçalves
Date: 2024-09-30 00:00:00
Tasks: Diversity, Domain Adaptation, Fairness, Multi-class Classification, Selection bias

Fairness in machine learning seeks to mitigate model bias against individuals based on sensitive features such as sex or age, often caused by an uneven representation of the population in the training data due to selection bias. Notably, bias unascribed to sensitive features is challenging to identify and typically goes undiagnosed, despite its prominence in complex high-dimensional data from fields like computer vision and molecular biomedicine. Strategies to mitigate unidentified bias and evaluate mitigation methods are crucially needed, yet remain underexplored. We introduce: (i) Diverse Class-Aware Self-Training (DCAST), model-agnostic mitigation aware of class-specific bias, which promotes sample diversity to counter confirmation bias of conventional self-training while leveraging unlabeled samples for an improved representation of the underlying population; (ii) hierarchy bias, multivariate and class-aware bias induction without prior knowledge. Models learned with DCAST showed improved robustness to hierarchy and other biases across eleven datasets, against conventional self-training and six prominent domain adaptation techniques. Advantage was largest on multi-class classification, emphasizing DCAST as a promising strategy for fairer learning in different contexts.
Robust LLM safeguarding via refusal feature adversarial training
Authors: Lei Yu, Virginie Do, Karen Hambardzumyan, Nicola Cancedda
Date: 2024-09-30 00:00:00

Large language models (LLMs) are vulnerable to adversarial attacks that can elicit harmful responses. Defending against such attacks remains challenging due to the opacity of jailbreaking mechanisms and the high computational cost of training LLMs robustly. We demonstrate that adversarial attacks share a universal mechanism for circumventing LLM safeguards that works by ablating a dimension in the residual stream embedding space called the refusal feature. We further show that the operation of refusal feature ablation (RFA) approximates the worst-case perturbation of offsetting model safety. Based on these findings, we propose Refusal Feature Adversarial Training (ReFAT), a novel algorithm that efficiently performs LLM adversarial training by simulating the effect of input-level attacks via RFA. Experiment results show that ReFAT significantly improves the robustness of three popular LLMs against a wide range of adversarial attacks, with considerably less computational overhead compared to existing adversarial training methods.
Inferring Thunderstorm Occurrence from Vertical Profiles of Convection-Permitting Simulations: Physical Insights from a Physical Deep Learning Model
Authors: Kianusch Vahid Yousefnia, Christoph Metzl, Tobias Bölle
Date: 2024-09-30 00:00:00

Thunderstorms have significant social and economic impacts due to heavy precipitation, hail, lightning, and strong winds, necessitating reliable forecasts. Thunderstorm forecasts based on numerical weather prediction (NWP) often rely on single-level surrogate predictors, like convective available potential energy and convective inhibition, derived from vertical profiles of three-dimensional atmospheric variables. In this study, we develop SALAMA 1D, a deep neural network which directly infers the probability of thunderstorm occurrence from vertical profiles of ten atmospheric variables, bypassing single-level predictors. By training the model on convection-permitting NWP forecasts, we allow SALAMA 1D to flexibly identify convective patterns, with the goal of enhancing forecast accuracy. The model's architecture is physically motivated: sparse connections encourage interactions at similar height levels while keeping model size and inference times computationally efficient, whereas a shuffling mechanism prevents the model from learning non-physical patterns tied to the vertical grid. SALAMA 1D is trained over Central Europe with lightning observations as the ground truth. Comparative analysis against a baseline machine learning model that uses single-level predictors shows SALAMA 1D's superior skill across various metrics and lead times of up to at least 11 hours. Moreover, expanding the archive of forecasts from which training examples are sampled improves skill, even when training set size remains constant. Finally, a sensitivity analysis using saliency maps indicates that our model relies on physically interpretable patterns consistent with established theoretical understanding, such as ice particle content near the tropopause, cloud cover, conditional instability, and low-level moisture.