pwc-archive/papers-with-abstracts huggingface dataset:
Permutation Equivariant Neural Functionals
Authors: Allan Zhou, KaiEn Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, J. Zico Kolter, Chelsea Finn
Date: 2023-02-27 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Inductive Bias

This work studies the design of neural networks that can process the weights or gradients of other neural networks, which we refer to as neural functional networks (NFNs). Despite a wide range of potential applications, including learned optimization, processing implicit neural representations, network editing, and policy evaluation, there are few unifying principles for designing effective architectures that process the weights of other networks. We approach the design of neural functionals through the lens of symmetry, in particular by focusing on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order. We introduce a framework for building permutation equivariant neural functionals, whose architectures encode these symmetries as an inductive bias. The key building blocks of this framework are NF-Layers (neural functional layers) that we constrain to be permutation equivariant through an appropriate parameter sharing scheme. In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks that require processing the weights of MLPs and CNNs, such as predicting classifier generalization, producing "winning ticket" sparsity masks for initializations, and classifying or editing implicit neural representations (INRs). In addition, we provide code for our models and experiments at https://github.com/AllanYangZhou/nfn.
SMAM: Self and Mutual Adaptive Matching for Skeleton-Based Few-Shot Action Recognition
Authors: Li Zhiheng; Gong Xuyuan; Song Ran; Duan Peng; Liu Jun; Zhang Wei
Date: 2022-12-27 00:00:00
Proceeding: IEEE Transactions on Image Processing 2022 12
Tasks: Action Recognition, Few-Shot action recognition, Few Shot Action Recognition

Abstract— This paper focuses on skeleton-based few-shot
action recognition. Since skeleton is essentially a sparse representation of human action, the feature maps extracted from it,
through a standard encoder network in the few-shot condition,
may not be sufficiently discriminative for some action sequences
that look partially similar to each other. To address this issue,
we propose a self and mutual adaptive matching (SMAM) module
to convert such feature maps into more discriminative feature
vectors. Our method, named as SMAM-Net, first leverages
both the temporal information associated with each individual
skeleton joint and the spatial relationship among them for feature
extraction. Then, the SMAM module adaptively measures the
similarity between labeled and query samples and further carries
out feature matching within the query set to distinguish similar
skeletons of various action categories. Experimental results show
that the SMAM-Net outperforms other baselines on the largescale NTU RGB + D 120 dataset in the tasks of one-shot and
five-shot action recognition. We also report our results on smaller
datasets including NTU RGB + D 60, SYSU and PKU-MMD to
demonstrate that our method is reliable and generalises well on
different datasets. Codes and the pretrained SMAM-Net will be
made publicly available.
Securing Healthcare with Deep Learning: A CNN-Based Model for medical IoT Threat Detection
Authors: Alireza Mohamadi, Hosna Ghahramani, Seyyed Amir Asghari, Mehdi Aminian
Date: 2024-10-26 00:00:00

The increasing integration of the Internet of Medical Things (IoMT) into healthcare systems has significantly enhanced patient care but has also introduced critical cybersecurity challenges. This paper presents a novel approach based on Convolutional Neural Networks (CNNs) for detecting cyberattacks within IoMT environments. Unlike previous studies that predominantly utilized traditional machine learning (ML) models or simpler Deep Neural Networks (DNNs), the proposed model leverages the capabilities of CNNs to effectively analyze the temporal characteristics of network traffic data. Trained and evaluated on the CICIoMT2024 dataset, which comprises 18 distinct types of cyberattacks across a range of IoMT devices, the proposed CNN model demonstrates superior performance compared to previous state-of-the-art methods, achieving a perfect accuracy of 99% in binary, categorical, and multiclass classification tasks. This performance surpasses that of conventional ML models such as Logistic Regression, AdaBoost, DNNs, and Random Forests. These findings highlight the potential of CNNs to substantially improve IoMT cybersecurity, thereby ensuring the protection and integrity of connected healthcare systems.
Criticality at work: scaling in the mouse cortex enhances performance
Authors: Gustavo G. Cambrainha, Daniel M. Castro, Nivaldo A. P. de Vasconcelos, Pedro Carelli, Mauro Copelli
Date: 2024-10-30 00:00:00

The critical brain hypothesis posits that neural systems operate near a phase transition, optimising the transmission, storage, and processing of information. While scale invariance and non-Gaussian dynamics - hallmarks of criticality - have been observed in brain activity, a direct link between criticality and behavioural performance remains unexplored. Here, we use a phenomenological renormalisation group (PRG) approach to examine neuronal activity in the primary visual cortex of mice performing a visual recognition task. We show that non-trivial scaling in neuronal activity is associated with enhanced task performance, with pronounced scaling observed during successful task completion. When rewards were removed or non-natural stimuli presented, these scaling signatures diminished. These results suggest that critical dynamics in the brain may be crucial for optimising behavioural outcomes, offering new insights into the functional role of criticality in cortical processing.
Vascular Segmentation of Functional Ultrasound Images using Deep Learning
Authors: Hana Sebia, Thomas Guyet, Mickaël Pereira, Marco Valdebenito, Hugues Berry, Benjamin Vidal
Date: 2024-10-28 00:00:00
Tasks: Deep Learning, Segmentation

Segmentation of medical images is a fundamental task with numerous applications. While MRI, CT, and PET modalities have significantly benefited from deep learning segmentation techniques, more recent modalities, like functional ultrasound (fUS), have seen limited progress. fUS is a non invasive imaging method that measures changes in cerebral blood volume (CBV) with high spatio-temporal resolution. However, distinguishing arterioles from venules in fUS is challenging due to opposing blood flow directions within the same pixel. Ultrasound localization microscopy (ULM) can enhance resolution by tracking microbubble contrast agents but is invasive, and lacks dynamic CBV quantification. In this paper, we introduce the first deep learning-based segmentation tool for fUS images, capable of differentiating signals from different vascular compartments, based on ULM automatic annotation and enabling dynamic CBV quantification. We evaluate various UNet architectures on fUS images of rat brains, achieving competitive segmentation performance, with 90% accuracy, a 71% F1 score, and an IoU of 0.59, using only 100 temporal frames from a fUS stack. These results are comparable to those from tubular structure segmentation in other imaging modalities. Additionally, models trained on resting-state data generalize well to images captured during visual stimulation, highlighting robustness. This work offers a non-invasive, cost-effective alternative to ULM, enhancing fUS data interpretation and improving understanding of vessel function. Our pipeline shows high linear correlation coefficients between signals from predicted and actual compartments in both cortical and deeperregions, showcasing its ability to accurately capture blood flow dynamics.
Accelerating Augmentation Invariance Pretraining
Authors: Jinhong Lin, Cheng-En Wu, Yibing Wei, Pedro Morgado
Date: 2024-10-27 00:00:00
Tasks: Contrastive Learning, Self-Supervised Learning

Our work tackles the computational challenges of contrastive learning methods, particularly for the pretraining of Vision Transformers (ViTs). Despite the effectiveness of contrastive learning, the substantial computational resources required for training often hinder their practical application. To mitigate this issue, we propose an acceleration framework, leveraging ViT's unique ability to generalize across inputs of varying sequence lengths. Our method employs a mix of sequence compression strategies, including randomized token dropout and flexible patch scaling, to reduce the cost of gradient estimation and accelerate convergence. We further provide an in-depth analysis of the gradient estimation error of various acceleration strategies as well as their impact on downstream tasks, offering valuable insights into the trade-offs between acceleration and performance. We also propose a novel procedure to identify an optimal acceleration schedule to adjust the sequence compression ratios to the training progress, ensuring efficient training without sacrificing downstream performance. Our approach significantly reduces computational overhead across various self-supervised learning algorithms on large-scale datasets. In ImageNet, our method achieves speedups of 4$\times$ in MoCo, 3.3$\times$ in SimCLR, and 2.5$\times$ in DINO, demonstrating substantial efficiency gains.
Mathematical Derivation Graphs: A Task for Summarizing Equation Dependencies in STEM Manuscripts
Authors: Vishesh Prasad, Brian Kim, Nickvash Kani
Date: 2024-10-26 00:00:00
Tasks: Articles

Recent advances in natural language processing (NLP), particularly with the emergence of large language models (LLMs), have significantly enhanced the field of textual analysis. However, while these developments have yielded substantial progress in analyzing textual data, applying analysis to mathematical equations and their relationships within texts has produced mixed results. In this paper, we take the initial steps toward understanding the dependency relationships between mathematical expressions in STEM articles. Our dataset, sourced from a random sampling of the arXiv corpus, contains an analysis of 107 published STEM manuscripts whose inter-equation dependency relationships have been hand-labeled, resulting in a new object we refer to as a derivation graph that summarizes the mathematical content of the manuscript. We exhaustively evaluate analytical and NLP-based models to assess their capability to identify and extract the derivation relationships for each article and compare the results with the ground truth. Our comprehensive testing finds that both analytical and NLP models (including LLMs) achieve $\sim$40-50% F1 scores for extracting derivation graphs from articles, revealing that the recent advances in NLP have not made significant inroads in comprehending mathematical texts compared to simpler analytic models. While current approaches offer a solid foundation for extracting mathematical information, further research is necessary to improve accuracy and depth in this area.
Moral Agency in Silico: Exploring Free Will in Large Language Models
Authors: Morgan S. Porter
Date: 2024-10-28 00:00:00
Tasks: Decision Making, Ethics, Philosophy

This study investigates the potential of deterministic systems, specifically large language models (LLMs), to exhibit the functional capacities of moral agency and compatibilist free will. We develop a functional definition of free will grounded in Dennett's compatibilist framework, building on an interdisciplinary theoretical foundation that integrates Shannon's information theory, Dennett's compatibilism, and Floridi's philosophy of information. This framework emphasizes the importance of reason-responsiveness and value alignment in determining moral responsibility rather than requiring metaphysical libertarian free will. Shannon's theory highlights the role of processing complex information in enabling adaptive decision-making, while Floridi's philosophy reconciles these perspectives by conceptualizing agency as a spectrum, allowing for a graduated view of moral status based on a system's complexity and responsiveness. Our analysis of LLMs' decision-making in moral dilemmas demonstrates their capacity for rational deliberation and their ability to adjust choices in response to new information and identified inconsistencies. Thus, they exhibit features of a moral agency that align with our functional definition of free will. These results challenge traditional views on the necessity of consciousness for moral responsibility, suggesting that systems with self-referential reasoning capacities can instantiate degrees of free will and moral reasoning in artificial and biological contexts. This study proposes a parsimonious framework for understanding free will as a spectrum that spans artificial and biological systems, laying the groundwork for further interdisciplinary research on agency and ethics in the artificial intelligence era.
Parameterized TDOA: Instantaneous TDOA Estimation and Localization for Mobile Targets in a Time-Division Broadcast Positioning System
Authors: Chenxin Tu, Xiaowei Cui, Gang Liu, Sihao Zhao, Mingquan Lu
Date: 2024-10-31 00:00:00
Tasks: Autonomous Navigation, parameter estimation

In a time-division broadcast positioning system (TDBPS), localizing mobile targets using classical time difference of arrival (TDOA) methods poses significant challenges. Concurrent TDOA measurements are infeasible because targets receive signals from different anchors and extract their transmission times at different reception times, as well as at varying positions. Traditional TDOA estimation schemes implicitly assume that the target remains stationary during the measurement period, which is impractical for mobile targets exhibiting high dynamics. Existing methods for mobile target localization are mostly specialized and rely on motion modeling and do not rely on the concurrent TDOA measurements. This issue limits their direct use of the well-established classical TDOA-based localization methods and complicating the entire localization process. In this paper, to obtain concurrent TDOA estimates at any instant out of the sequential measurements for direct use of existing TDOA-based localization methods, we propose a novel TDOA estimation method, termed parameterized TDOA (P-TDOA). By approximating the time-varying TDOA as a polynomial function over a short period, we transform the TDOA estimation problem into a model parameter estimation problem and derive the desired TDOA estimates thereafter. Theoretical analysis shows that, under certain conditions, the proposed P-TDOA method closely approaches the Cramer-Rao Lower Bound (CRLB) for TDOA estimation in concurrent measurement scenarios, despite measurements being obtained sequentially. Extensive numerical simulations validate our theoretical analysis and demonstrate the effectiveness of the proposed method, highlighting substantial improvements over existing approaches across various scenarios.
Reasons and Solutions for the Decline in Model Performance after Editing
Authors: Xiusheng Huang, Jiaxiang Liu, Yequan Wang, Kang Liu
Date: 2024-10-31 00:00:00
Tasks: knowledge editing

Knowledge editing technology has received widespread attention for low-cost updates of incorrect or outdated knowledge in large-scale language models. However, recent research has found that edited models often exhibit varying degrees of performance degradation. The reasons behind this phenomenon and potential solutions have not yet been provided. In order to investigate the reasons for the performance decline of the edited model and optimize the editing method, this work explores the underlying reasons from both data and model perspectives. Specifically, 1) from a data perspective, to clarify the impact of data on the performance of editing models, this paper first constructs a Multi-Question Dataset (MQD) to evaluate the impact of different types of editing data on model performance. The performance of the editing model is mainly affected by the diversity of editing targets and sequence length, as determined through experiments. 2) From a model perspective, this article explores the factors that affect the performance of editing models. The results indicate a strong correlation between the L1-norm of the editing model layer and the editing accuracy, and clarify that this is an important factor leading to the bottleneck of editing performance. Finally, in order to improve the performance of the editing model, this paper further proposes a Dump for Sequence (D4S) method, which successfully overcomes the previous editing bottleneck by reducing the L1-norm of the editing layer, allowing users to perform multiple effective edits and minimizing model damage. Our code is available at https://github.com/nlpkeg/D4S.
Ada-MSHyper: Adaptive Multi-Scale Hypergraph Transformer for Time Series Forecasting
Authors: Zongjiang Shang, Ling Chen, Binqing Wu, Dongliang Cui
Date: 2024-10-31 00:00:00
Tasks: Time Series, Time Series Forecasting

Although transformer-based methods have achieved great success in multi-scale temporal pattern interaction modeling, two key challenges limit their further development: (1) Individual time points contain less semantic information, and leveraging attention to model pair-wise interactions may cause the information utilization bottleneck. (2) Multiple inherent temporal variations (e.g., rising, falling, and fluctuating) entangled in temporal patterns. To this end, we propose Adaptive Multi-Scale Hypergraph Transformer (Ada-MSHyper) for time series forecasting. Specifically, an adaptive hypergraph learning module is designed to provide foundations for modeling group-wise interactions, then a multi-scale interaction module is introduced to promote more comprehensive pattern interactions at different scales. In addition, a node and hyperedge constraint mechanism is introduced to cluster nodes with similar semantic information and differentiate the temporal variations within each scales. Extensive experiments on 11 real-world datasets demonstrate that Ada-MSHyper achieves state-of-the-art performance, reducing prediction errors by an average of 4.56%, 10.38%, and 4.97% in MSE for long-range, short-range, and ultra-long-range time series forecasting, respectively. Code is available at https://github.com/shangzongjiang/Ada-MSHyper.
Bridging Geometric States via Geometric Diffusion Bridge
Authors: Shengjie Luo, Yixian Xu, Di He, Shuxin Zheng, Tie-Yan Liu, LiWei Wang
Date: 2024-10-31 00:00:00

The accurate prediction of geometric state evolution in complex systems is critical for advancing scientific domains such as quantum chemistry and material modeling. Traditional experimental and computational methods face challenges in terms of environmental constraints and computational demands, while current deep learning approaches still fall short in terms of precision and generality. In this work, we introduce the Geometric Diffusion Bridge (GDB), a novel generative modeling framework that accurately bridges initial and target geometric states. GDB leverages a probabilistic approach to evolve geometric state distributions, employing an equivariant diffusion bridge derived by a modified version of Doob's $h$-transform for connecting geometric states. This tailored diffusion process is anchored by initial and target geometric states as fixed endpoints and governed by equivariant transition kernels. Moreover, trajectory data can be seamlessly leveraged in our GDB framework by using a chain of equivariant diffusion bridges, providing a more detailed and accurate characterization of evolution dynamics. Theoretically, we conduct a thorough examination to confirm our framework's ability to preserve joint distributions of geometric states and capability to completely model the underlying dynamics inducing trajectory distributions with negligible error. Experimental evaluations across various real-world scenarios show that GDB surpasses existing state-of-the-art approaches, opening up a new pathway for accurately bridging geometric states and tackling crucial scientific challenges with improved accuracy and applicability.
Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use
Authors: Jiajun Xi, Yinong He, Jianing Yang, Yinpei Dai, Joyce Chai
Date: 2024-10-31 00:00:00
Tasks: Diversity, Informativeness, Reinforcement Learning (RL)

In real-world scenarios, it is desirable for embodied agents to have the ability to leverage human language to gain explicit or implicit knowledge for learning tasks. Despite recent progress, most previous approaches adopt simple low-level instructions as language inputs, which may not reflect natural human communication. It's not clear how to incorporate rich language use to facilitate task learning. To address this question, this paper studies different types of language inputs in facilitating reinforcement learning (RL) embodied agents. More specifically, we examine how different levels of language informativeness (i.e., feedback on past behaviors and future guidance) and diversity (i.e., variation of language expressions) impact agent learning and inference. Our empirical results based on four RL benchmarks demonstrate that agents trained with diverse and informative language feedback can achieve enhanced generalization and fast adaptation to new tasks. These findings highlight the pivotal role of language use in teaching embodied agents new tasks in an open world. Project website: https://github.com/sled-group/Teachable_RL
Understanding Optimization in Deep Learning with Central Flows
Authors: Jeremy M. Cohen, Alex Damian, Ameet Talwalkar, Zico Kolter, Jason D. Lee
Date: 2024-10-31 00:00:00
Tasks: Deep Learning, Navigate

Optimization in deep learning remains poorly understood, even in the simple setting of deterministic (i.e. full-batch) training. A key difficulty is that much of an optimizer's behavior is implicitly determined by complex oscillatory dynamics, referred to as the "edge of stability." The main contribution of this paper is to show that an optimizer's implicit behavior can be explicitly captured by a "central flow:" a differential equation which models the time-averaged optimization trajectory. We show that these flows can empirically predict long-term optimization trajectories of generic neural networks with a high degree of numerical accuracy. By interpreting these flows, we reveal for the first time 1) the precise sense in which RMSProp adapts to the local loss landscape, and 2) an "acceleration via regularization" mechanism, wherein adaptive optimizers implicitly navigate towards low-curvature regions in which they can take larger steps. This mechanism is key to the efficacy of these adaptive optimizers. Overall, we believe that central flows constitute a promising tool for reasoning about optimization in deep learning.