pwc-archive/papers-with-abstracts huggingface dataset:
Efficient Training of Physics-enhanced Neural ODEs via Direct Collocation and Nonlinear Programming
Authors: Linus Langenkamp, Philip Hannebohm, Bernhard Bachmann
Date: 2025-05-06 00:00:00

We propose a novel approach for training Physics-enhanced Neural ODEs (PeNODEs) by expressing the training process as a dynamic optimization problem. The full model, including neural components, is discretized using a high-order implicit Runge-Kutta method with flipped Legendre-Gauss-Radau points, resulting in a large-scale nonlinear program (NLP) efficiently solved by state-of-the-art NLP solvers such as Ipopt. This formulation enables simultaneous optimization of network parameters and state trajectories, addressing key limitations of ODE solver-based training in terms of stability, runtime, and accuracy. Extending on a recent direct collocation-based method for Neural ODEs, we generalize to PeNODEs, incorporate physical constraints, and present a custom, parallelized, open-source implementation. Benchmarks on a Quarter Vehicle Model and a Van-der-Pol oscillator demonstrate superior accuracy, speed, and generalization with smaller networks compared to other training techniques. We also outline a planned integration into OpenModelica to enable accessible training of Neural DAEs.
Small-Scale-Fading-Aware Resource Allocation in Wireless Federated Learning
Authors: Jiacheng Wang, Le Liang, Hao Ye, Chongtao Guo, Shi Jin
Date: 2025-05-06 00:00:00
Tasks: Federated Learning, Multi-agent Reinforcement Learning

Judicious resource allocation can effectively enhance federated learning (FL) training performance in wireless networks by addressing both system and statistical heterogeneity. However, existing strategies typically rely on block fading assumptions, which overlooks rapid channel fluctuations within each round of FL gradient uploading, leading to a degradation in FL training performance. Therefore, this paper proposes a small-scale-fading-aware resource allocation strategy using a multi-agent reinforcement learning (MARL) framework. Specifically, we establish a one-step convergence bound of the FL algorithm and formulate the resource allocation problem as a decentralized partially observable Markov decision process (Dec-POMDP), which is subsequently solved using the QMIX algorithm. In our framework, each client serves as an agent that dynamically determines spectrum and power allocations within each coherence time slot, based on local observations and a reward derived from the convergence analysis. The MARL setting reduces the dimensionality of the action space and facilitates decentralized decision-making, enhancing the scalability and practicality of the solution. Experimental results demonstrate that our QMIX-based resource allocation strategy significantly outperforms baseline methods across various degrees of statistical heterogeneity. Additionally, ablation studies validate the critical importance of incorporating small-scale fading dynamics, highlighting its role in optimizing FL performance.
Coding for Computation: Efficient Compression of Neural Networks for Reconfigurable Hardware
Authors: Hans Rosenberger, Rodrigo Fischer, Johanna S. Fröhlich, Ali Bereyhi, Ralf R. Müller
Date: 2025-04-24 00:00:00

As state of the art neural networks (NNs) continue to grow in size, their resource-efficient implementation becomes ever more important. In this paper, we introduce a compression scheme that reduces the number of computations required for NN inference on reconfigurable hardware such as FPGAs. This is achieved by combining pruning via regularized training, weight sharing and linear computation coding (LCC). Contrary to common NN compression techniques, where the objective is to reduce the memory used for storing the weights of the NNs, our approach is optimized to reduce the number of additions required for inference in a hardware-friendly manner. The proposed scheme achieves competitive performance for simple multilayer perceptrons, as well as for large scale deep NNs such as ResNet-34.
Faster MoE LLM Inference for Extremely Large Models
Authors: Haoqi Yang, Luohe Shi, Qiwei Li, Zuchao Li, Ping Wang, Bo Du, Mengjia Shen, Hai Zhao
Date: 2025-05-06 00:00:00
Tasks: Inference Optimization, Mixture-of-Experts

Sparse Mixture of Experts (MoE) large language models (LLMs) are gradually becoming the mainstream approach for ultra-large-scale models. Existing optimization efforts for MoE models have focused primarily on coarse-grained MoE architectures. With the emergence of DeepSeek Models, fine-grained MoE models are gaining popularity, yet research on them remains limited. Therefore, we want to discuss the efficiency dynamic under different service loads. Additionally, fine-grained models allow deployers to reduce the number of routed experts, both activated counts and total counts, raising the question of how this reduction affects the trade-off between MoE efficiency and performance. Our findings indicate that while deploying MoE models presents greater challenges, it also offers significant optimization opportunities. Reducing the number of activated experts can lead to substantial efficiency improvements in certain scenarios, with only minor performance degradation. Reducing the total number of experts provides limited efficiency gains but results in severe performance degradation. Our method can increase throughput by at least 10\% without any performance degradation. Overall, we conclude that MoE inference optimization remains an area with substantial potential for exploration and improvement.
Rate-Distortion-Perception Theory for the Quadratic Wasserstein Space
Authors: Xiqiang Qu, Jun Chen, Lei Yu, Xiangyu Xu
Date: 2025-04-24 00:00:00

We establish a single-letter characterization of the fundamental distortion-rate-perception tradeoff with limited common randomness under the squared error distortion measure and the squared Wasserstein-2 perception measure. Moreover, it is shown that this single-letter characterization can be explicitly evaluated for the Gaussian source. Various notions of universal representation are also clarified.
Causal Intervention Framework for Variational Auto Encoder Mechanistic Interpretability
Authors: Dip Roy
Date: 2025-05-06 00:00:00
Tasks: Disentanglement, Specificity

Mechanistic interpretability of deep learning models has emerged as a crucial research direction for understanding the functioning of neural networks. While significant progress has been made in interpreting discriminative models like transformers, understanding generative models such as Variational Autoencoders (VAEs) remains challenging. This paper introduces a comprehensive causal intervention framework for mechanistic interpretability of VAEs. We develop techniques to identify and analyze "circuit motifs" in VAEs, examining how semantic factors are encoded, processed, and disentangled through the network layers. Our approach uses targeted interventions at different levels: input manipulations, latent space perturbations, activation patching, and causal mediation analysis. We apply our framework to both synthetic datasets with known causal relationships and standard disentanglement benchmarks. Results show that our interventions can successfully isolate functional circuits, map computational graphs to causal graphs of semantic factors, and distinguish between polysemantic and monosemantic units. Furthermore, we introduce metrics for causal effect strength, intervention specificity, and circuit modularity that quantify the interpretability of VAE components. Experimental results demonstrate clear differences between VAE variants, with FactorVAE achieving higher disentanglement scores (0.084) and effect strengths (mean 4.59) compared to standard VAE (0.064, 3.99) and Beta-VAE (0.051, 3.43). Our framework advances the mechanistic understanding of generative models and provides tools for more transparent and controllable VAE architectures.
Uncovering the Limitations of Model Inversion Evaluation -- Benchmarks and Connection to Type-I Adversarial Attacks
Authors: Sy-Tuyen Ho, Koh Jun Hao, Ngoc-Bao Nguyen, Alexander Binder, Ngai-Man Cheung
Date: 2025-05-06 00:00:00

Model Inversion (MI) attacks aim to reconstruct information of private training data by exploiting access to machine learning models. The most common evaluation framework for MI attacks/defenses relies on an evaluation model that has been utilized to assess progress across almost all MI attacks and defenses proposed in recent years. In this paper, for the first time, we present an in-depth study of MI evaluation. Firstly, we construct the first comprehensive human-annotated dataset of MI attack samples, based on 28 setups of different MI attacks, defenses, private and public datasets. Secondly, using our dataset, we examine the accuracy of the MI evaluation framework and reveal that it suffers from a significant number of false positives. These findings raise questions about the previously reported success rates of SOTA MI attacks. Thirdly, we analyze the causes of these false positives, design controlled experiments, and discover the surprising effect of Type I adversarial features on MI evaluation, as well as adversarial transferability, highlighting a relationship between two previously distinct research areas. Our findings suggest that the performance of SOTA MI attacks has been overestimated, with the actual privacy leakage being significantly less than previously reported. In conclusion, we highlight critical limitations in the widely used MI evaluation framework and present our methods to mitigate false positive rates. We remark that prior research has shown that Type I adversarial attacks are very challenging, with no existing solution. Therefore, we urge to consider human evaluation as a primary MI evaluation framework rather than merely a supplement as in previous MI research. We also encourage further work on developing more robust and reliable automatic evaluation frameworks.
Path Integral Methods for Synthesizing and Preventing Stealthy Attacks in Nonlinear Cyber-Physical Systems
Authors: Apurva Patil, Kyle Morgenstein, Luis Sentis, Takashi Tanaka
Date: 2025-04-23 00:00:00

This paper studies the synthesis and mitigation of stealthy attacks in nonlinear cyber-physical systems (CPS). To quantify stealthiness, we employ the Kullback-Leibler (KL) divergence, a measure rooted in hypothesis testing and detection theory, which captures the trade-off between an attacker's desire to remain stealthy and her goal of degrading system performance. First, we synthesize the worst-case stealthy attack in nonlinear CPS using the path integral approach. Second, we consider how a controller can mitigate the impact of such stealthy attacks by formulating a minimax KL control problem, yielding a zero-sum game between the attacker and the controller. Again, we leverage a path integral-based solution that computes saddle-point policies for both players through Monte Carlo simulations. We validate our approach using unicycle navigation and cruise control problems, demonstrating how an attacker can covertly drive the system into unsafe regions, and how the controller can adapt her policy to combat the worst-case attacks.
AnomalyMatch: Discovering Rare Objects of Interest with Semi-supervised and Active Learning
Authors: Pablo Gómez, David O'Ryan
Date: 2025-05-06 00:00:00
Tasks: Active Learning, Anomaly Detection, Astronomy, Binary Classification, GPU

Anomaly detection in large datasets is essential in fields such as astronomy and computer vision; however, supervised methods typically require extensive anomaly labelling, which is often impractical. We present AnomalyMatch, an anomaly detection framework combining the semi-supervised FixMatch algorithm using EfficientNet classifiers with active learning. By treating anomaly detection as a semi-supervised binary classification problem, we efficiently utilise limited labelled and abundant unlabelled images. We allow iterative model refinement in a user interface for expert verification of high-confidence anomalies and correction of false positives. Built for astronomical data, AnomalyMatch generalises readily to other domains facing similar data challenges. Evaluations on the GalaxyMNIST astronomical dataset and the miniImageNet natural-image benchmark under severe class imbalance (1% anomalies for miniImageNet) display strong performance: starting from five to ten labelled anomalies and after three active learning cycles, we achieve an average AUROC of 0.95 (miniImageNet) and 0.86 (GalaxyMNIST), with respective AUPRC of 0.77 and 0.71. After active learning cycles, anomalies are ranked with 71% (miniImageNet) to 93% precision in the 1% of the highest-ranked images. AnomalyMatch is tailored for large-scale applications, efficiently processing predictions for 100 million images within three days on a single GPU. Integrated into ESAs Datalabs platform, AnomalyMatch facilitates targeted discovery of scientifically valuable anomalies in vast astronomical datasets. Our results underscore the exceptional utility and scalability of this approach for anomaly discovery, highlighting the value of specialised approaches for domains characterised by severe label scarcity.
Relationship between Hölder Divergence and Functional Density Power Divergence: Intersection and Generalization
Authors: Masahiro Kobayashi
Date: 2025-04-23 00:00:00
Tasks: scoring rule

In this study, we discuss the relationship between two families of density-power-based divergences with functional degrees of freedom -- the H\"{o}lder divergence and the functional density power divergence (FDPD) -- based on their intersection and generalization. These divergence families include the density power divergence and the $\gamma$-divergence as special cases. First, we prove that the intersection of the H\"{o}lder divergence and the FDPD is limited to a general divergence family introduced by Jones et al. (Biometrika, 2001). Subsequently, motivated by the fact that H\"{o}lder's inequality is used in the proofs of nonnegativity for both the H\"{o}lder divergence and the FDPD, we define a generalized divergence family, referred to as the $\xi$-H\"{o}lder divergence. The nonnegativity of the $\xi$-H\"{o}lder divergence is established through a combination of the inequalities used to prove the nonnegativity of the H\"{o}lder divergence and the FDPD. Furthermore, we derive an inequality between the composite scoring rules corresponding to different FDPDs based on the $\xi$-H\"{o}lder divergence. Finally, we prove that imposing the mathematical structure of the H\"{o}lder score on a composite scoring rule results in the $\xi$-H\"{o}lder divergence.
Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization
Authors: Yu-Han Wu, Pierre Marion, Gérard Biau, Claire Boyer
Date: 2025-02-05 00:00:00
Tasks: Denoising, Memorization

Denoising score matching plays a pivotal role in the performance of diffusion-based generative models. However, the empirical optimal score--the exact solution to the denoising score matching--leads to memorization, where generated samples replicate the training data. Yet, in practice, only a moderate degree of memorization is observed, even without explicit regularization. In this paper, we investigate this phenomenon by uncovering an implicit regularization mechanism driven by large learning rates. Specifically, we show that in the small-noise regime, the empirical optimal score exhibits high irregularity. We then prove that, when trained by stochastic gradient descent with a large enough learning rate, neural networks cannot stably converge to a local minimum with arbitrarily small excess risk. Consequently, the learned score cannot be arbitrarily close to the empirical optimal score, thereby mitigating memorization. To make the analysis tractable, we consider one-dimensional data and two-layer neural networks. Experiments validate the crucial role of the learning rate in preventing memorization, even beyond the one-dimensional setting.
I-Con: A Unifying Framework for Representation Learning
Authors: Shaden Alshammari, John Hershey, Axel Feldmann, William T. Freeman, Mark Hamilton
Date: 2025-04-23 00:00:00
Tasks: Contrastive Learning, Dimensionality Reduction, Representation Learning

As the field of representation learning grows, there has been a proliferation of different loss functions to solve different classes of problems. We introduce a single information-theoretic equation that generalizes a large collection of modern loss functions in machine learning. In particular, we introduce a framework that shows that several broad classes of machine learning methods are precisely minimizing an integrated KL divergence between two conditional distributions: the supervisory and learned representations. This viewpoint exposes a hidden information geometry underlying clustering, spectral methods, dimensionality reduction, contrastive learning, and supervised learning. This framework enables the development of new loss functions by combining successful techniques from across the literature. We not only present a wide array of proofs, connecting over 23 different approaches, but we also leverage these theoretical results to create state-of-the-art unsupervised image classifiers that achieve a +8% improvement over the prior state-of-the-art on unsupervised classification on ImageNet-1K. We also demonstrate that I-Con can be used to derive principled debiasing methods which improve contrastive representation learners.
Efficient Multivariate Time Series Forecasting via Calibrated Language Models with Privileged Knowledge Distillation
Authors: Chenxi Liu, Hao Miao, Qianxiong Xu, Shaowen Zhou, Cheng Long, Yan Zhao, Ziyue Li, Rui Zhao
Date: 2025-05-04 00:00:00
Tasks: Knowledge Distillation, Multivariate Time Series Forecasting, Time Series, Time Series Forecasting

Multivariate time series forecasting (MTSF) endeavors to predict future observations given historical data, playing a crucial role in time series data management systems. With advancements in large language models (LLMs), recent studies employ textual prompt tuning to infuse the knowledge of LLMs into MTSF. However, the deployment of LLMs often suffers from low efficiency during the inference phase. To address this problem, we introduce TimeKD, an efficient MTSF framework that leverages the calibrated language models and privileged knowledge distillation. TimeKD aims to generate high-quality future representations from the proposed cross-modality teacher model and cultivate an effective student model. The cross-modality teacher model adopts calibrated language models (CLMs) with ground truth prompts, motivated by the paradigm of Learning Under Privileged Information (LUPI). In addition, we design a subtractive cross attention (SCA) mechanism to refine these representations. To cultivate an effective student model, we propose an innovative privileged knowledge distillation (PKD) mechanism including correlation and feature distillation. PKD enables the student to replicate the teacher's behavior while minimizing their output discrepancy. Extensive experiments on real data offer insight into the effectiveness, efficiency, and scalability of the proposed TimeKD.
Uplink Sum Rate Maximization for Pinching Antenna-Assisted Multiuser MISO
Authors: Jiarui Zhang, Hao Xu, Chongjun Ouyang, Qiuyun Zou, Hongwen Yang
Date: 2025-04-23 00:00:00

This article investigates the application of pinching-antenna systems (PASS) in multiuser multiple-input single-output (MISO) communications. Two sum-rate maximization problems are formulated under minimum mean square error (MMSE) decoding, with and without successive interference cancellation (SIC). To address the joint optimization of pinching antenna locations and user transmit powers, a fractional programming-based approach is proposed. Numerical results validate the effectiveness of the proposed method and show that PASS can significantly enhance uplink sum-rate performance compared to conventional fixed-antenna designs.
Modeling Musical Genre Trajectories through Pathlet Learning
Authors: Lilian Marey, Charlotte Laclau, Bruno Sguerra, Tiphaine Viard, Manuel Moussallam
Date: 2025-05-06 00:00:00
Tasks: Dictionary Learning, Diversity, Recommendation Systems

The increasing availability of user data on music streaming platforms opens up new possibilities for analyzing music consumption. However, understanding the evolution of user preferences remains a complex challenge, particularly as their musical tastes change over time. This paper uses the dictionary learning paradigm to model user trajectories across different musical genres. We define a new framework that captures recurring patterns in genre trajectories, called pathlets, enabling the creation of comprehensible trajectory embeddings. We show that pathlet learning reveals relevant listening patterns that can be analyzed both qualitatively and quantitatively. This work improves our understanding of users' interactions with music and opens up avenues of research into user behavior and fostering diversity in recommender systems. A dataset of 2000 user histories tagged by genre over 17 months, supplied by Deezer (a leading music streaming company), is also released with the code.
LiDAL-Assisted RLNC-NOMA in OWC Systems
Authors: Ahmed A. Hassan, Ahmad Adnan Qidan, Taisir Elgorashi, Jaafar Elmirghani
Date: 2025-04-23 00:00:00

Optical wireless communication (OWC) is envisioned as a key enabler for immersive indoor data transmission in future wireless communication networks. However, multi-user interference management arises as a challenge in dense indoor OWC systems composed of multiple optical access points (APs) serving multiple users. In this paper, we propose a novel dual-function OWC system for communication and localization. Non-orthogonal multiple access (NOMA) with random linear network coding (RLNC) is designed for data transmission, where NOMA allows the serving of multiple users simultaneously through controlling the power domain, and RLNC helps minimize errors that might occur during signal processing phase. This setup is assisted with a light detection and localization system (LiDAL) that can passively obtain spatio-temporal indoor information of user presence and location for dynamic-user grouping. The designed LiDAL system helps to improve the estimation of channel state information (CSI) in realistic indoor network scenarios, where the CSI of indoor users might be noisy and/or highly correlated. We evaluate the performance of NOMA combined with RLNC by analyzing the probability of successful decoding compared to conventional NOMA and orthogonal schemes. In addition, we derive the Cramer-Rao Lower Bound (CRLB) to evaluate the accuracy of location estimation. The results show that the proposed RLNC-NOMA improves the probability of successful decoding and the overall system performance. The results also show the high accuracy of the unbiased location estimator and its assistant in reducing the imperfection of CSI, leading to high overall system performance.