pwc-archive/papers-with-abstracts huggingface dataset:
Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Efficient Weight Quantization
Authors: Lucas Deckers, Benjamin Vandersmissen, Ing Jyh Tsang, Werner van Leekwijck, Steven Latré
Date: 2024-09-24 00:00:00
Tasks: Knowledge Distillation, Quantization

The proliferation of Artificial Neural Networks (ANNs) has led to increased energy consumption, raising concerns about their sustainability. Spiking Neural Networks (SNNs), which are inspired by biological neural systems and operate using sparse, event-driven spikes to communicate information between neurons, offer a potential solution due to their lower energy requirements. An alternative technique for reducing a neural network's footprint is quantization, which compresses weight representations to decrease memory usage and energy consumption. In this study, we present Twin Network Augmentation (TNA), a novel training framework aimed at improving the performance of SNNs while also facilitating an enhanced compression through low-precision quantization of weights. TNA involves co-training an SNN with a twin network, optimizing both networks to minimize their cross-entropy losses and the mean squared error between their output logits. We demonstrate that TNA significantly enhances classification performance across various vision datasets and in addition is particularly effective when applied when reducing SNNs to ternary weight precision. Notably, during inference , only the ternary SNN is retained, significantly reducing the network in number of neurons, connectivity and weight size representation. Our results show that TNA outperforms traditional knowledge distillation methods and achieves state-of-the-art performance for the evaluated network architecture on benchmark datasets, including CIFAR-10, CIFAR-100, and CIFAR-10-DVS. This paper underscores the effectiveness of TNA in bridging the performance gap between SNNs and ANNs and suggests further exploration into the application of TNA in different network architectures and datasets.
Data Poisoning-based Backdoor Attack Framework against Supervised Learning Rules of Spiking Neural Networks
Authors: Lingxin Jin, Meiyu Lin, Wei Jiang, Jinyu Zhan
Date: 2024-09-24 00:00:00
Tasks: Backdoor Attack, Data Poisoning

Spiking Neural Networks (SNNs), the third generation neural networks, are known for their low energy consumption and high robustness. SNNs are developing rapidly and can compete with Artificial Neural Networks (ANNs) in many fields. To ensure that the widespread use of SNNs does not cause serious security incidents, much research has been conducted to explore the robustness of SNNs under adversarial sample attacks. However, many other unassessed security threats exist, such as highly stealthy backdoor attacks. Therefore, to fill the research gap in this and further explore the security vulnerabilities of SNNs, this paper explores the robustness performance of SNNs trained by supervised learning rules under backdoor attacks. Specifically, the work herein includes: i) We propose a generic backdoor attack framework that can be launched against the training process of existing supervised learning rules and covers all learnable dataset types of SNNs. ii) We analyze the robustness differences between different learning rules and between SNN and ANN, which suggests that SNN no longer has inherent robustness under backdoor attacks. iii) We reveal the vulnerability of conversion-dependent learning rules caused by backdoor migration and further analyze the migration ability during the conversion process, finding that the backdoor migration rate can even exceed 99%. iv) Finally, we discuss potential countermeasures against this kind of backdoor attack and its technical challenges and point out several promising research directions.
A Spacetime Perspective on Dynamical Computation in Neural Information Processing Systems
Authors: T. Anderson Keller, Lyle Muller, Terrence J. Sejnowski, Max Welling
Date: 2024-09-20 00:00:00

There is now substantial evidence for traveling waves and other structured spatiotemporal recurrent neural dynamics in cortical structures; but these observations have typically been difficult to reconcile with notions of topographically organized selectivity and feedforward receptive fields. We introduce a new 'spacetime' perspective on neural computation in which structured selectivity and dynamics are not contradictory but instead are complimentary. We show that spatiotemporal dynamics may be a mechanism by which natural neural systems encode approximate visual, temporal, and abstract symmetries of the world as conserved quantities, thereby enabling improved generalization and long-term working memory.
Stimulus-to-Stimulus Learning in RNNs with Cortical Inductive Biases
Authors: Pantelis Vafidis, Antonio Rangel
Date: 2024-09-20 00:00:00
Tasks: Inductive Bias

Animals learn to predict external contingencies from experience through a process of conditioning. A natural mechanism for conditioning is stimulus substitution, whereby the neuronal response to a stimulus with no prior behavioral significance becomes increasingly identical to that generated by a behaviorally significant stimulus it reliably predicts. We propose a recurrent neural network model of stimulus substitution which leverages two forms of inductive bias pervasive in the cortex: representational inductive bias in the form of mixed stimulus representations, and architectural inductive bias in the form of two-compartment pyramidal neurons that have been shown to serve as a fundamental unit of cortical associative learning. The properties of these neurons allow for a biologically plausible learning rule that implements stimulus substitution, utilizing only information available locally at the synapses. We show that the model generates a wide array of conditioning phenomena, and can learn large numbers of associations with an amount of training commensurate with animal experiments, without relying on parameter fine-tuning for each individual experimental task. In contrast, we show that commonly used Hebbian rules fail to learn generic stimulus-stimulus associations with mixed selectivity, and require task-specific parameter fine-tuning. Our framework highlights the importance of multi-compartment neuronal processing in the cortex, and showcases how it might confer cortical animals the evolutionary edge.
Scalable Multi-Objective Optimization for Robust Traffic Signal Control in Uncertain Environments
Authors: Weian Guo, Wuzhao Li, Zhiou Zhang, Lun Zhang, Li Li, Dongyang Li
Date: 2024-09-20 00:00:00
Tasks: Traffic Signal Control

Intelligent traffic signal control is essential to modern urban management, with important impacts on economic efficiency, environmental sustainability, and quality of daily life. However, in current decades, it continues to pose significant challenges in managing large-scale traffic networks, coordinating intersections, and ensuring robustness under uncertain traffic conditions. This paper presents a scalable multi-objective optimization approach for robust traffic signal control in dynamic and uncertain urban environments. A multi-objective optimization model is proposed in this paper, which incorporates stochastic variables and probabilistic traffic patterns to capture traffic flow dynamics and uncertainty. We propose an algorithm named Adaptive Hybrid Multi-Objective Optimization Algorithm (AHMOA), which addresses the uncertainties of city traffic, including network-wide signal coordination, fluctuating patterns, and environmental impacts. AHMOA simultaneously optimizes multiple objectives, such as average delay, network stability, and system robustness, while adapting to unpredictable changes in traffic. The algorithm combines evolutionary strategies with an adaptive mechanism to balance exploration and exploitation, and incorporates a memory-based evaluation mechanism to leverage historical traffic data. Simulations are conducted in different cities including Manhattan, Paris, Sao Paulo, and Istanbul. The experimental results demonstrate that AHMOA consistently outperforms several state-of-the-art algorithms and the algorithm is competent to provide scalable, robust Pareto optimal solutions for managing complex traffic systems under uncertain environments.
Exploring the Performance-Reproducibility Trade-off in Quality-Diversity
Authors: Manon Flageat, Hannah Janmohamed, Bryan Lim, Antoine Cully
Date: 2024-09-20 00:00:00
Tasks: Diversity

Quality-Diversity (QD) algorithms have exhibited promising results across many domains and applications. However, uncertainty in fitness and behaviour estimations of solutions remains a major challenge when QD is used in complex real-world applications. While several approaches have been proposed to improve the performance in uncertain applications, many fail to address a key challenge: determining how to prioritise solutions that perform consistently under uncertainty, in other words, solutions that are reproducible. Most prior methods improve fitness and reproducibility jointly, ignoring the possibility that they could be contradictory objectives. For example, in robotics, solutions may reliably walk at 90% of the maximum velocity in uncertain environments, while solutions that walk faster are also more prone to falling over. As this is a trade-off, neither one of these two solutions is "better" than the other. Thus, algorithms cannot intrinsically select one solution over the other, but can only enforce given preferences over these two contradictory objectives. In this paper, we formalise this problem as the performance-reproducibility trade-off for uncertain QD. We propose four new a-priori QD algorithms that find optimal solutions for given preferences over the trade-offs. We also propose an a-posteriori QD algorithm for when these preferences cannot be defined in advance. Our results show that our approaches successfully find solutions that satisfy given preferences. Importantly, by simply accounting for this trade-off, our approaches perform better than existing uncertain QD methods. This suggests that considering the performance-reproducibility trade-off unlocks important stepping stones that are usually missed when only performance is optimised.
Emergent Collective Reproduction via Evolving Neuronal Flocks
Authors: Nam H. Le, Richard Watson, Mike Levin, Chrys Buckley
Date: 2024-09-20 00:00:00
Tasks: Artificial Life

This study facilitates the understanding of evolutionary transitions in individuality (ETIs) through a novel artificial life framework, named VitaNova, that intricately merges self-organization and natural selection to simulate the emergence of complex, reproductive groups. By dynamically modelling individual agents within an environment that challenges them with predators and spatial constraints, VitaNova elucidates the mechanisms by which simple agents evolve into cohesive units exhibiting collective reproduction. The findings underscore the synergy between self-organized behaviours and adaptive evolutionary strategies as fundamental drivers of ETIs. This approach not only contributes to a deeper understanding of higher-order biological individuality but also sets a new precedent in the empirical investigation of ETIs, challenging and extending current theoretical frameworks.
DS2TA: Denoising Spiking Transformer with Attenuated Spatiotemporal Attention
Authors: Boxun Xu, Hejia Geng, Yuxuan Yin, Peng Li
Date: 2024-09-20 00:00:00
Tasks: Denoising

Vision Transformers (ViT) are current high-performance models of choice for various vision applications. Recent developments have given rise to biologically inspired spiking transformers that thrive in ultra-low power operations on neuromorphic hardware, however, without fully unlocking the potential of spiking neural networks. We introduce DS2TA, a Denoising Spiking transformer with attenuated SpatioTemporal Attention, designed specifically for vision applications. DS2TA introduces a new spiking attenuated spatiotemporal attention mechanism that considers input firing correlations occurring in both time and space, thereby fully harnessing the computational power of spiking neurons at the core of the transformer architecture. Importantly, DS2TA facilitates parameter-efficient spatiotemporal attention computation without introducing extra weights. DS2TA employs efficient hashmap-based nonlinear spiking attention denoisers to enhance the robustness and expressive power of spiking attention maps. DS2TA demonstrates state-of-the-art performances on several widely adopted static image and dynamic neuromorphic datasets. Operated over 4 time steps, DS2TA achieves 94.92% top-1 accuracy on CIFAR10 and 77.47% top-1 accuracy on CIFAR100, as well as 79.1% and 94.44% on CIFAR10-DVS and DVS-Gesture using 10 time steps.
Evolutionary Algorithms for One-Sided Bipartite Crossing Minimisation
Authors: Jakob Baumann, Ignaz Rutter, Dirk Sudholt
Date: 2024-09-05 00:00:00
Tasks: Evolutionary Algorithms

Evolutionary algorithms (EAs) are universal solvers inspired by principles of natural evolution. In many applications, EAs produce astonishingly good solutions. As they are able to deal with complex optimisation problems, they show great promise for hard problems encountered in the field of graph drawing.To complement recent theoretical advances in the analysis of EAs on graph drawing, we contribute a fundamental empirical study. We consider the so-called \textsc{One-Sided Bipartite Crossing Minimisation (OBCM)}: given two layers of a bipartite graph and a fixed horizontal order of vertices on the first layer, the task is to order the vertices on the second layer to minimise the number of edge crossings. We empirically analyse the performance of simple EAs for OBCM and compare different mutation operators on the underlying permutation ordering problem: exchanging two elements (\textit{exchange}), swapping adjacent elements (\textit{swap}) and jumping an element to a new position (\textit{jump}). EAs using jumps easily outperform all deterministic algorithms in terms of solution quality after a reasonable number of generations. We also design variations of the best-performing EAs to reduce the execution time for each generation. The improved EAs can obtain the same solution quality as before and run up to 100 times faster.
Learning Task-Based Trainable Neuromorphic ADCs via Power-Aware Distillation
Authors: Tal Vol, Loai Danial, Nir Shlezinger
Date: 2024-09-04 00:00:00
Tasks: Quantization

The ability to process signals in digital form depends on analog-to-digital converters (ADCs). Traditionally, ADCs are designed to ensure that the digital representation closely matches the analog signal. However, recent studies have shown that significant power and memory savings can be achieved through task-based acquisition, where the acquisition process is tailored to the downstream processing task. An emerging technology for task-based acquisition involves the use of memristors, which are considered key enablers for neuromorphic computing. Memristors can implement ADCs with tunable mappings, allowing adaptation to specific system tasks or power constraints. In this work, we study task-based acquisition for a generic classification task using memristive ADCs. We consider the unique characteristics of this such neuromorphic ADCs, including their power consumption and noisy read-write behavior, and propose a physically compliant model based on resistive successive approximation register ADCs integrated with memristor components, enabling the adjustment of quantization regions. To optimize performance, we introduce a data-driven algorithm that jointly tunes task-based memristive ADCs alongside both digital and analog processing. Our design addresses the inherent stochasticity of memristors through power-aware distillation, complemented by a specialized learning algorithm that adapts to their unique analog-to-digital mapping. The proposed approach is shown to enhance accuracy by up to 27% and reduce power consumption by up to 66% compared to uniform ADCs. Even under noisy conditions, our method achieves substantial gains, with accuracy improvements of up to 19% and power reductions of up to 57%. These results highlight the effectiveness of our power-aware neuromorphic ADCs in improving system performance across diverse tasks.
Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model
Authors: Kaiwen Tang, Zhanglu Yan, Weng-Fai Wong
Date: 2024-09-04 00:00:00
Tasks: Knowledge Distillation, Language Modeling, Language Modelling, Quantization

For reasons such as privacy, there are use cases for language models at the edge. This has given rise to small language models (SLMs) targeted for deployment in resource-constrained devices where energy efficiency is a significant concern. Spiking neural networks (SNNs) offer a promising solution due to their energy efficiency, and there are already works on realizing transformer-based models on SNNs. However, key operations like softmax and layer normalization (LN) are difficult to implement on neuromorphic hardware, and many of these early works sidestepped them. To address these challenges, we introduce Sorbet, a transformer-based spiking language model that is more neuromorphic hardware-compatible. Sorbet incorporates a novel shifting-based softmax called PTsoftmax and a power normalization method using bit-shifting (BSPN), both designed to replace the respective energy-intensive operations. By leveraging knowledge distillation and model quantization, Sorbet achieved a highly compressed binary weight model that maintains competitive performance while significantly reducing energy consumption. We validate Sorbet's effectiveness through extensive testing on the GLUE benchmark and a series of ablation studies, demonstrating its potential as an energy-efficient solution for language model inference.
An Efficient General-Purpose Optical Accelerator for Neural Networks
Authors: Sijie Fei, Amro Eldebiky, Grace Li Zhang, Bing Li, Ulf Schlichtmann
Date: 2024-09-02 00:00:00

General-purpose optical accelerators (GOAs) have emerged as a promising platform to accelerate deep neural networks (DNNs) due to their low latency and energy consumption. Such an accelerator is usually composed of a given number of interleaving Mach-Zehnder- Interferometers (MZIs). This interleaving architecture, however, has a low efficiency when accelerating neural networks of various sizes due to the mismatch between weight matrices and the GOA architecture. In this work, a hybrid GOA architecture is proposed to enhance the mapping efficiency of neural networks onto the GOA. In this architecture, independent MZI modules are connected with microring resonators (MRRs), so that they can be combined to process large neural networks efficiently. Each of these modules implements a unitary matrix with inputs adjusted by tunable coefficients. The parameters of the proposed architecture are searched using genetic algorithm. To enhance the accuracy of neural networks, selected weight matrices are expanded to multiple unitary matrices applying singular value decomposition (SVD). The kernels in neural networks are also adjusted to use up the on-chip computational resources. Experimental results show that with a given number of MZIs, the mapping efficiency of neural networks on the proposed architecture can be enhanced by 21.87%, 21.20%, 24.69%, and 25.52% for VGG16 and Resnet18 on datasets Cifar10 and Cifar100, respectively. The energy consumption and computation latency can also be reduced by over 67% and 21%, respectively.
Trust-Region Sequential Quadratic Programming for Stochastic Optimization with Random Models
Authors: Yuchen Fang, Sen Na, Michael W. Mahoney, Mladen Kolar
Date: 2024-09-24 00:00:00
Tasks: Stochastic Optimization

In this work, we consider solving optimization problems with a stochastic objective and deterministic equality constraints. We propose a Trust-Region Sequential Quadratic Programming method to find both first- and second-order stationary points. Our method utilizes a random model to represent the objective function, which is constructed from stochastic observations of the objective and is designed to satisfy proper adaptive accuracy conditions with a high but fixed probability. To converge to first-order stationary points, our method computes a gradient step in each iteration defined by minimizing a quadratic approximation of the objective subject to a (relaxed) linear approximation of the problem constraints and a trust-region constraint. To converge to second-order stationary points, our method additionally computes an eigen step to explore the negative curvature of the reduced Hessian matrix, as well as a second-order correction step to address the potential Maratos effect, which arises due to the nonlinearity of the problem constraints. Such an effect may impede the method from moving away from saddle points. Both gradient and eigen step computations leverage a novel parameter-free decomposition of the step and the trust-region radius, accounting for the proportions among the feasibility residual, optimality residual, and negative curvature. We establish global almost sure first- and second-order convergence guarantees for our method, and present computational results on CUTEst problems, regression problems, and saddle-point problems to demonstrate its superiority over existing line-search-based stochastic methods.
Quantum Machine Learning in Drug Discovery: Applications in Academia and Pharmaceutical Industries
Authors: Anthony M. Smaldone, Yu Shee, Gregory W. Kyro, Chuzhi Xu, Nam P. Vu, Rishab Dutta, Marwa H. Farag, Alexey Galda, Sandeep Kumar, Elica Kyoseva, Victor S. Batista
Date: 2024-09-24 00:00:00
Tasks: Drug Discovery, Molecular Property Prediction, Property Prediction, Quantum Machine Learning

The nexus of quantum computing and machine learning - quantum machine learning - offers the potential for significant advancements in chemistry. This review specifically explores the potential of quantum neural networks on gate-based quantum computers within the context of drug discovery. We discuss the theoretical foundations of quantum machine learning, including data encoding, variational quantum circuits, and hybrid quantum-classical approaches. Applications to drug discovery are highlighted, including molecular property prediction and molecular generation. We provide a balanced perspective, emphasizing both the potential benefits and the challenges that must be addressed.
Generalization vs. Specialization under Concept Shift
Authors: Alex Nguyen, David J. Schwab, Vudtiwat Ngampruetikorn
Date: 2024-09-23 00:00:00

Machine learning models are often brittle under distribution shift, i.e., when data distributions at test time differ from those during training. Understanding this failure mode is central to identifying and mitigating safety risks of mass adoption of machine learning. Here we analyze ridge regression under concept shift -- a form of distribution shift in which the input-label relationship changes at test time. We derive an exact expression for prediction risk in the high-dimensional limit. Our results reveal nontrivial effects of concept shift on generalization performance, depending on the properties of robust and nonrobust features of the input. We show that test performance can exhibit a nonmonotonic data dependence, even when double descent is absent. Finally, our experiments on MNIST and FashionMNIST suggest that this intriguing behavior is present also in classification problems.