Original: N ETWORK INSENSITIVITY TO PARAMETER NOISE VIA PARAMETER ATTACK DURING TRAINING
Revised: N ETWORK INSENSITIVITY TO PARAMETER NOISE VIA ADVERSARIAL REGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neuromorphic neural network processors, in the form of compute-in-memory crossbar arrays of memristors, or in the form of subthreshold analog and mixed-signal
Revised: Neuromorphic neural network processors, in the form of compute-in-memory crossbar arrays of memristors, or in the form of subthreshold analog and mixed-signal

Original: ASICs, promise enormous advantages in compute density and energy efﬁciency for NN-based ML tasks.
Revised: ASICs, promise enormous advantages in compute density and energy efﬁciency for NN-based ML tasks.

Original: However, these technologies are prone to computational non-idealities, due to process variation and intrinsic device physics.
Revised: However, these technologies are prone to computational non-idealities, due to process variation and intrinsic device physics.

Original: This degrades the task performance of networks deployed to the processor, by introducing parameter noise into the deployed model.
Revised: This degrades the task performance of networks deployed to the processor, by introducing parameter noise into the deployed model.

Original: While it is possible to calibrate each device, or train networks individually for each processor, these approaches are expensive and impractical for commercial deployment.
Revised: While it is possible to calibrate each device, or train networks individually for each processor, these approaches are expensive and impractical for commercial deployment.

Original: Alternative methods are therefore needed to train networks that are inherently robust against parameter variation, as a consequence of network architecture and parameters.
Revised: Alternative methods are therefore needed to train networks that are inherently robust against parameter variation, as a consequence of network architecture and parameters.

Original: We present a new network training algorithm that attacks network parameters during training, and promotes robust performance during inference in the face of random parameter variation.
Revised: We present a new network training algorithm that attacks network parameters during training and promotes robust performance during inference in the face of random parameter variation.

Original: Our approach introduces a loss regularization term that penalizes the susceptibility of a network to weight perturbation.
Revised: Our approach introduces a loss regularization term that penalizes the susceptibility of a network to weight perturbation.

Original: We compare against previous approaches for producing parameter insensitivity such as dropout, weight smoothing and introducing parameter noise during training.
Revised: We compare against previous approaches for producing parameter insensitivity such as dropout, weight smoothing and introducing parameter noise during training.

Original: We show that our approach produces models that are more robust to random mismatch-induced parameter variation as well as to targeted parameter variation.
Revised: We show that our approach produces models that are more robust to random mismatch-induced parameter variation as well as to targeted parameter variation.

Original: Our approach ﬁnds minima in ﬂatter locations in the weight-loss landscape compared with other approaches, highlighting that the networks found by our technique are less sensitive to parameter perturbation.
Revised: Our approach ﬁnds minima in ﬂatter locations in the weight-loss landscape compared with other approaches, highlighting that the networks found by our technique are less sensitive to parameter perturbation.

Original: Our work provides an approach to deploy neural network architectures to inference devices that suffer from computational non-idealities, with minimal loss of performance.
Revised: Our work provides an approach to deploy neural network architectures to inference devices that suffer from computational non-idealities, with minimal loss of performance.

Original: This method will enable deployment at scale to novel energy-efﬁcient computational substrates, promoting cheaper and more prevalent edge inference.
Revised: This method will enable deployment at scale to novel energy-efﬁcient computational substrates, promoting cheaper and more prevalent edge inference

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is increasing interest in NN and ML inference on IoT and embedded devices, which imposes energy constraints due to small battery capacity and untethered operation.
Revised: There is increasing interest in NN and ML inference on IoT and embedded devices, which imposes energy constraints due to small battery capacity and untethered operation.

Original: Existing edge inference solutions based on CPUs or vector processing engines such as GPUs or TPUs are improving in energy efﬁciency, but still entail considerable energy cost (Huang et al., 2009).
Revised: Existing edge inference solutions based on CPUs or vector processing engines such as GPUs or TPUs are improving in energy efﬁciency, but still entail considerable energy cost (Huang et al., 2009).

Original: Alternative compute architectures such as memristor crossbar arrays and mixed-signal event-driven neural network accelerators promise signiﬁcantly reduced energy consumption for edge inference tasks.
Revised: Alternative compute architectures such as memristor crossbar arrays and mixed-signal event-driven neural network accelerators promise signiﬁcantly reduced energy consumption for edge inference tasks.

Original: Novel non-volatile memory technologies such as resistive RAM and phase-change materials (Chen, 2016; Yu & Chen, 2016) promise increased memory density with multiple bits per memory cell, as well as compact compute-in-memory for NN inference tasks (Verma et al., 2019).
Revised: Novel non-volatile memory technologies such as resistive RAM and phase-change materials (Chen, 2016; Yu & Chen, 2016) promise increased memory density with multiple bits per memory cell, as well as compact compute-in-memory for NN inference tasks (Verma et al., 2019).

Original: Analog implementations of neurons and synapses, coupled with asynchronous digital routing fabrics, permit high sparsity in both network architecture and activity, thereby reducing energy costs associated with computation.
Revised: Analog implementations of neurons and synapses, coupled with asynchronous digital routing fabrics, permit high sparsity in both network architecture and activity, thereby reducing energy costs associated with computation.

Original: However, both of these novel compute fabrics introduce complexity in the form of computational non-idealities, which do not exist for pure synchronous digital solutions.
Revised: However, both of these novel compute fabrics introduce complexity in the form of computational non-idealities, which do not exist for pure synchronous digital solutions.

Original: Some novel memory technologies support several bits per memory cell, but with uncertainty about the precise value stored on each cycle (Le Gallo et al., 2018; Wu et al., 2019).
Revised: Some novel memory technologies support several bits per memory cell, but with uncertainty about the precise value stored on each cycle (Le Gallo et al., 2018b; Wu et al., 2019).

Original: Others exhibit signiﬁcant drift in stored states (Joshi et al., 2020).
Revised: Others exhibit signiﬁcant drift in stored states (Joshi et al., 2020).

Original: Inference processors based on analog and mixed-signal devices (Neckar et al., 2019; Moradi et al., 2018; Cassidy et al., 2016; Schemmel et al., 2010) exhibit parameter variation across the surface of a chip, and between chips, due to manufacturing process non-idealities.
Revised: Inference processors based on analog and mixed-signal devices (Neckar et al., 2019; Moradi et al., 2018; Cassidy et al., 2016; Schemmel et al., 2010) exhibit parameter variation across the surface of a chip, and between chips, due to manufacturing process non-idealities.

Original: Collectively these processes known as “device mismatch” manifest as frozen parameter noise in weights and neuron parameters.
Revised: Collectively these processes known as “device mismatch” manifest as frozen parameter noise in weights and neuron parameters.

Original: In all cases the mismatch between conﬁgured and implemented network parameters degrades the task performance by modifying the resulting mapping between input and output.
Revised: In all cases the mismatch between conﬁgured and implemented network parameters degrades the task performance by modifying the resulting mapping between input and output.

Original: Existing solutions for deploying networks to inference devices that exhibit mismatch mostly focus on per-device calibration or re-training (Ambrogio et al., 2018; Bauer et al., 2019; Nandakumar et al., 2020).
Revised: Existing solutions for deploying networks to inference devices that exhibit mismatch mostly focus on per-device calibration or re-training (Ambrogio et al., 2018; Bauer et al., 2019; Nandakumar et al., 2020a).

Original: However, this, and other approaches such as few-shot learning or meta learning entail signiﬁcant per-device handling costs, making them unﬁt for commercial deployment.
Revised: However, this, and other approaches such as few-shot learning or meta learning entail signiﬁcant per-device handling costs, making them unﬁt for commercial deployment.

Original: We consider a network to be “robust” if the output of a network to a given input does not change in the face of parameter perturbation.
Revised: We consider a network to be “robust” if the output of a network to a given input does not change in the face of parameter perturbation.

Original: With this goal, network architectures that are intrinsically robust against device mismatch can be investigated (Thakur et al., 2018; Büchel et al., 2021).
Revised: With this goal, network architectures that are intrinsically robust against device mismatch can be investigated (Thakur et al., 2018; Büchel et al., 2021).

Original: Another approach is to introduce parameter perturbations during training that promote robustness during inference, for example via random pruning (dropout) (Srivastava et al., 2014) or by injecting noise (Murray & Edwards, 1994; Joshi et al., 2020).
Revised: Another approach is to introduce parameter perturbations during training that promote robustness during inference, for example via random pruning (dropout) (Srivastava et al., 2014) or by injecting noise (Murray & Edwards, 1994).

Original: In this paper we introduce a novel solution, by applying adversarial training approaches to parameter mismatch.
Revised: In this paper we introduce a novel solution, by applying adversarial training approaches to parameter mismatch.

Original: Most existing adversarial training methods attack the input space.
Revised: Most existing adversarial training methods attack the input space.

Original: Here we describe an adversarial attack during training that seeks the parameter perturbation that causes the maximum degradation in network response.
Revised: Here we describe an adversarial attack during training that seeks the parameter perturbation that causes the maximum degradation in network response.

Original: In summary, we make the following contributions:
Revised: In summary, we make the following contributions:

Original: • We propose a novel algorithm for gradient-based supervised training of networks that are robust against parameter mismatch, by performing adversarial training in theweight space.
Revised: • We propose a novel algorithm for gradient-based supervised training of networks that are robust against parameter mismatch, by performing adversarial training in the

Original:  • We demonstrate that our algorithm ﬂattens the weight-loss landscape and therefore leads to models that are inherently more robust to parameter noise.
Revised: weight space. • We demonstrate that our algorithm ﬂattens the weight-loss landscape and therefore leads to models that are inherently more robust to parameter noise.

Original: • We show that our approach outperforms existing methods in terms of robustness.
Revised: • We show that our approach outperforms existing methods in terms of robustness.

Revised: • We validate our algorithm on a highly accurate Phase Change Memory (PCM)-based Computein-Memory (CiM) simulator and achieve new state-of-the-art results in terms of performance and performance retention over

Revised: time.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Research to date has focused mainly on adversarial attacks in the input space.
Revised: Research to date has focused mainly on adversarial attacks in the input space.

Original: With an increasing number of adversarial attacks, an increasing number of schemes defending against those attacks have been proposed (Wang et al., 2020; Zhang et al., 2019; Madry et al., 2019; Moosavi-Dezfooli et al., 2018).
Revised: With an increasing number of adversarial attacks, an increasing number of schemes defending against those attacks have been proposed (Wang et al., 2020; Zhang et al., 2019; Madry et al., 2019; Moosavi-Dezfooli et al., 2018).

Original: In contrast, adversarial attacks in parameter space have received little attention.
Revised: In contrast, adversarial attacks in parameter space have received little attention.

Original: Where parameter-space adversaries have been examined, it has been to enhance performance in semisupervised learning (Cicek & Soatto, 2019) or to improve robustness to input-space adversarial attacks (Wu et al.,  2020).
Revised: Where parameter-space adversaries have been examined, it has been to enhance performance in semisupervised learning (Cicek & Soatto, 2019), to improve robustness to input-space adversarial attacks (Wu et al., 2020), or to improve generalisation capability (Zheng et al., 2020).

Original: We deﬁne “robustness” to mean that the network output should change only minimally in the face of a parameter perturbation — in other words, the weight-loss landscape should be as ﬂat as possible at a loss minimum.
Revised: We deﬁne “robustness” to mean that the network output should change only minimally in the face of a parameter perturbation — in other words, the weight-loss landscape should be as ﬂat as possible at a loss minimum.

Original: Other algorithms that promote ﬂat loss landscapes may therefore also be useful to promote robustness to parameter perturbations.
Revised: Other algorithms that promote ﬂat loss landscapes may therefore also be useful to promote robustness to parameter perturbations.

Original: Dropout (Srivastava et al., 2014) is a widely used method to reduce overﬁtting.
Revised: Dropout (Srivastava et al., 2014) is a widely used method to reduce overﬁtting.

Original: During training, a random subset of units are chosen with some probability, and these units are pruned from the network for a single trial or batch.
Revised: During training, a random subset of units are chosen with some probability, and these units are pruned from the network for a single trial or batch.

Original: This results in the network learning to distribute its computation across many units, and acts as a regularization against overﬁtting.
Revised: This results in the network learning to distribute its computation across many units, and acts as a regularization against overﬁtting.

Original: Entropy-SGD (Chaudhari et al., 2019) is a network optimisation method that minimises the local entropy around a solution in parameter space.
Revised: Entropy-SGD (Chaudhari et al., 2019) is a network optimisation method that minimises the local entropy around a solution in parameter space.

Original: This results in a smoothed parameter-loss landscape that should penalize sharp minima.
Revised: This results in a smoothed parameter-loss landscape that should penalize sharp minima.

Original: Adversarial Block Coordinate Descent (ABCD) (Cicek & Soatto, 2019) was proposed in order to complement input-space smoothing with weight-space smoothing in semi-supervised learning.
Revised: Adversarial Block Coordinate Descent (ABCD) (Cicek & Soatto, 2019) was proposed in order to complement input-space smoothing with weight-space smoothing in semi-supervised learning.

Original: ABCD repeatedly picks half of the network weights and performs one step of gradient ascent on them, followed by applying gradient descent on the other half.
Revised: ABCD repeatedly picks half of the network weights and performs one step of gradient ascent on them, followed by applying gradient descent on the other half.

Original: Adversarial Weight Perturbation (AWP) (Wu et al., 2020) was designed to improve the robustness of a network to adversarial attacks in the input space.
Revised: Adversarial Weight Perturbation (AWP) (Wu et al., 2020) was designed to improve the robustness of a network to adversarial attacks in the input space.

Original: The authors use Projected Gradient Ascent (PGA) on the network parameters to approximate a worst case perturbation of the weights Θ (cid:48) .
Revised: The authors use Projected Gradient Ascent (PGA) on the network parameters to approximate a worst case perturbation of the weights Θ (cid:48) .

Original: PGA repeatedly computes the gradient of a loss function and updates the parameters in the direction of the (positive) gradient.
Revised: PGA repeatedly computes the gradient of a loss function and updates the parameters in the direction of the (positive) gradient.

Original: After each update, the parameters are projected back onto a ball (e.g. in l 2 ) around the original parameters to ensure that  maximum distance is kept.
Revised: After each update, the parameters are projected back onto a ball (e.g. in l 2 ) around the original parameters to ensure that a maximum distance is kept.

Original: Having identiﬁed an adversarial perturbation in the weight-space, an adversarial perturbation in the input-space is also found using PGA.
Revised: Having identiﬁed an adversarial perturbation in the weight-space, an adversarial perturbation in the input-space is also found using PGA.

Original: Finally, the original weights Θ are updated using the gradient of the loss evaluated at the adversarial perturbation Θ (cid:48) .
Revised: Finally, the original weights Θ are updated using the gradient of the loss evaluated at the adversarial perturbation Θ (cid:48) .

Revised: Adversarial Model Perturbation (AMP) (Zheng et al., 2020) improves the generalisation of conventional neural networks by optimizing a standard loss evaluated using parameters that were perturbed adversarially using PGA.

Revised: Unlike our method, (Zheng et al., 2020) did not formulate the loss function as a trade-off between performance and robustness.

Revised: Furthermore, the presented algorithm, unlike our method, treats the perturbation ∆ Θ to the parameters Θ as a constant during backpropagation.

Original: TRadeoff-inspired Adversarial DEfense via Surrogate-loss minimization (TRADES) (Zhang et al., 2019) is a method for training networks that are robust against adversarial examples in the input space.
Revised: TRadeoff-inspired Adversarial DEfense via Surrogate-loss minimization (TRADES) (Zhang et al., 2019) is a method for training networks that are robust against adversarial examples in the input space.

Original: The method consists of adding a boundary loss term to the loss function that measures how the network performance changes when the input is attacked.
Revised: The method consists of adding a boundary loss term to the loss function that measures how the network performance changes when the input is attacked.

Original: The boundary loss does not take the labels into account, so scaling it by a factor β  allows for a principled trade-off between the robustness and the accuracy of the network.
Revised: The boundary loss does not take the labels into account, so scaling it by a factor β rob allows for a principled trade-off between the robustness and the accuracy of the network.

Original: Noise injection during the forward pass (Murray & Edwards, 1994) is a simple method for increasing network robustness to parameter noise.
Revised: Noise injection during the forward pass (Murray & Edwards, 1994) is a simple method for increasing network robustness to parameter noise.

Original: This method adds Gaussian noise to the network parameters during the forward pass and computes weight gradients with respect to the original parameters.
Revised: This method adds Gaussian noise to the network parameters during the forward pass and computes weight gradients with respect to the original parameters.

Original: This method regularizes the gradient magnitudes of output units with respect to the weights, thus enforcing distributed information processing and insensitivity to parameter noise.
Revised: This method regularizes the gradient magnitudes of output units with respect to the weights, thus enforcing distributed information processing and insensitivity to parameter noise.

Original: We refer to this method as “Forward Noise”.
Revised: We refer to this method as “Forward Noise”.

Revised: In a recent paper (Stutz et al., 2021), a method for improving the resilience to random and targeted bit errors in SRAM cells on digital Deep Neural Network (DNN) accelerators was proposed.

Revised: By employing adversarial or random bit ﬂips during training, the authors signiﬁcantly improved the robustness to bit perturbations, enabling the accelerators to be operated below the conventional supply voltage.

Original:  M ETHODS
Revised:  M ETHODS

Original: We use Θ to denote the set of parameters of a neural network f ( x, Θ) that are trainable and susceptible to mismatch.
Revised: We use Θ to denote the set of parameters of a neural network f ( x, Θ) that are trainable and susceptible to mismatch.

Original: The adversarial weights are denoted Θ ∗ , where Θ ∗ t are the adversarial weights at the t -th iteration of PGA.
Revised: The adversarial weights are denoted Θ ∗ , where Θ ∗ t are the adversarial weights at the t -th iteration of PGA.

Original: We denote the PGA-adversary as a function A that maps parameters Θ to attacking parameters Θ ∗ .
Revised: We denote the PGA-adversary as a function A that maps parameters Θ to attacking parameters Θ ∗ .

Original: We denote a mini-batch of training examples as X with y being the corresponding ground-truth labels.
Revised: We denote a mini-batch of training examples as X with y being the corresponding ground-truth labels.

Original: (cid:81) E pζ ( m ) denotes the projection operator on the ζ -ellipsoid in l p space.
Revised: (cid:81) E pζ ( m ) denotes the projection operator on the ζ -ellipsoid in l p space.

Original: The operator (cid:12) denotes elementwise multiplication.
Revised: The operator (cid:12) denotes elementwise multiplication.

Original: As shown in (Joshi et al., 2020), the effect of component mismatch on a network parameter can be modelled using a Gaussian distribution where the standard deviation dependslinearly on the parameter magnitude.
Revised: As shown in (Büchel et al., 2021; Joshi et al., 2020), the effect of component mismatch on a network parameter can be modelled using a Gaussian distribution where the standard deviation depends on the parameter magnitude.

Original: For complex Spiking Neural Networks (SNNs), “network parameters” can refer to additional quantities such as neuronal and synaptic time constants or spiking thresholds.
Revised: For complex Spiking Neural Networks (SNNs), “network parameters” can refer to additional quantities such as neuronal and synaptic time constants or spiking thresholds.

Original: Our training approach can be equally applied to these additional parameters.
Revised: Our training approach can be equally applied to these additional parameters.

Original: However, in this paper we restrict ourselves to perturbations in the network weights for simplicity.
Revised: However, in this paper we restrict ourselves to perturbations in the network weights for simplicity.

Original: We deﬁne the value of an individual parameter when deployed on a neuromorphic chip as
Revised: We deﬁne the value of an individual parameter when deployed on a neuromorphic chip as

Original: [Equation]
Revised: [Equation]

Original:  where ζ governs the perturbation magnitude, referred to as the “mismatch level”.
Revised: (1) where ζ governs the perturbation magnitude, referred to as the “mismatch level”.

Original: The physics underlying the neuronal- and synaptic circuits lead to a model where the amount of noise introduced into the system depends linearly on the magnitude of the parameters.
Revised: The physics underlying the neuronal- and synaptic circuits lead to a model where the amount of noise introduced into the system depends linearly on the magnitude of the parameters.

Original: If mismatch-induced perturbations had constant standard deviation independent of weight values, one could use the weight-scale invariance of neural networks as a means to achieve robustness, by simply scaling up all network weights (See
Revised: If mismatch-induced perturbations had constant standard deviation independent of weight values, one could use the weight-scale invariance of neural networks as a means to achieve robustness, by simply scaling up all network weights (see

Original: Fig. S4).
Revised: Figure S4).

Original: The linear dependence of weight magnitude and mismatch noise precludes this approach.
Revised: The linear dependence of weight magnitude and mismatch noise precludes this approach.

Original: In contrast to adversarial attacks in the input space (Carlini & Wagner, 2016; Moosavi-Dezfooli et al., 2015; Madry et al., 2019; Goodfellow et al., 2015), our method relies on adversarial attacks in parameter space .
Revised: In contrast to adversarial attacks in the input space (Carlini & Wagner, 2016; Moosavi-Dezfooli et al., 2015; Madry et al., 2019; Goodfellow et al., 2015), our method relies on adversarial attacks in parameter space .

Original: During training, we approximate the worst case perturbation of the network parameters using PGA and update the network parameters in order to mitigate these attacks.
Revised: During training, we approximate the worst case perturbation of the network parameters using PGA and update the network parameters in order to mitigate these attacks.

Original: To trade-off robustness and performance, we use a surrogate loss (Zhang et al., 2019) to capture the difference in output between the normal and attacked network.
Revised: To trade-off robustness and performance, we use a surrogate loss (Zhang et al., 2019) to capture the difference in output between the normal and attacked network.

Original: Algorithm 1 illustrates the training procedure in more detail. 
Revised: Algorithm 1 illustrates the training procedure in more detail. begin [Equation]

Revised: for to N steps

Revised: [Equation]

Revised: end

Original: Unlike adversarial training in the input space, where adversarial inputs can be seen as a form of data augmentation, adversarial training in the parameter space poses the following challenge: Because the parameters that are attacked are the same parameters being optimized, performing gradient descent using the same loss that was used for PGA would simply revert the previous updates and no learning would occur.
Revised: [Equation] end Algorithm 1: In l ∞ , v corresponds to sign ( g ) and the step size α is | Θ |

Revised: [Equation]

Revised: pζ

Revised: [Equation]

Revised: m ) denotes the projection operator on the ζ attack -ellipsoid in l p space.

Revised: In l ∞ this corresponds to min(max( m, Θ − (cid:15) ) , Θ + (cid:15) ) with (cid:15)

Revised: =

Revised: ζ attack (cid:12) | Θ | .

Revised: ζ attack and β rob are hyperparameters of our model.

Revised: Unlike adversarial training in the input space, where adversarial inputs can be seen as a form of data augmentation, adversarial training in the parameter space poses the following challenge: Because the parameters that are attacked are the same parameters being optimized, performing gradient descent using the same loss that was used for PGA would simply revert the previous updates and no learning would occur.

Original: ABCD circumvents this problem by masking one half of the parameters in the adversarial loop and masking the other half during the gradient descent step.
Revised: ABCD circumvents this problem by masking one half of the parameters in the adversarial loop and masking the other half during the gradient descent step.

Original: However, this limits the adversary in its power, and requires multiple iterations to be performed in order to update all parameters at least once.
Revised: However, this limits the adversary in its power, and requires multiple iterations to be performed in order to update all parameters at least once.

Original: AWP approached this problem by assuming that the gradient of the loss with respect to the attacking parameters can be used in order to update the original parameters to favor minima in ﬂatter locations in weight-space.
Revised: AWP approached this problem by assuming that the gradient of the loss with respect to the attacking parameters can be used in order to update the original parameters to favor minima in ﬂatter locations in weight-space.

Original: However, it is not clear whether this assumption always holds since the gradient of the loss with respect to the attacking parameters is not necessarily the same direction that would lead to a ﬂatter region in the weight loss-landscape.
Revised: However, it is not clear whether this assumption always holds since the gradient of the loss with respect to the attacking parameters is not necessarily the same direction that would lead to a ﬂatter region in the weight loss-landscape.

Original: We approach this problem slightly differently:
Revised: We approach this problem slightly differently:

Original: Similar to the TRADES algorithm (Zhang et al., 2019), our algorithm optimizes a natural (task) loss and a separate robustness loss.
Revised: Similar to the TRADES algorithm (Zhang et al., 2019), our algorithm optimizes a natural (task) loss and a separate robustness loss.

Original: [Equation]
Revised: [Equation]

Original: Using a different loss for capturing the susceptibility of the network to adversarial attacks enables us to simultaneously optimise for performance and robustness, without PGA interfering with the gradient descent step.
Revised: Using a different loss for capturing the susceptibility of the network to adversarial attacks enables us to simultaneously optimise for performance and robustness, without PGA interfering with the gradient descent step.

Original: In our experiments, L Rob is deﬁned as
Revised: In our experiments, L rob is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: This formulation comes with a large computational overhead since it requires computing the Jacobian
Revised: This formulation comes with a large computational overhead since it requires computing the Jacobian

Original: J Θ ∗ (Θ) of a complex recurrent relation between Θ and Θ ∗ .
Revised: J Θ ∗ (Θ) of a complex recurrent relation between Θ and Θ ∗ .

Original: To make our algorithm more efﬁcient we assume that the Jacobian is diagonal, meaning that Θ ∗ =
Revised: To make our algorithm more efﬁcient we assume that the Jacobian is diagonal, meaning that Θ ∗ =

Original: Θ + ∆Θ for some ∆Θ given by the adversary.
Revised: Θ + ∆Θ for some ∆Θ given by the adversary.

Original: In l ∞ , the Jacobian can then be calculated efﬁciently using (see suppl.
Revised: In l ∞ , the Jacobian can then be calculated efﬁciently using (see suppl.

Original: material for details):
Revised: material for details):

Original: [Equation]
Revised: [Equation]

Original: By making this assumption, our algorithm effectively multiplies the original training time by the number of PGA steps, similar to (Wu et al., 2020; Cicek & Soatto, 2019).
Revised: By making this assumption, our algorithm effectively multiplies the original training time by the number of PGA steps, similar to (Wu et al., 2020; Cicek & Soatto, 2019; Zheng et al., 2020).

Original: Because component mismatch is independently proportional to the magnitude of each parameter, one has to model the space in which the adversary can search for a perturbation using an axis-aligned ellipsoid in l 2 and an axis-aligned box in l ∞ .
Revised: Because component mismatch is independently proportional to the magnitude of each parameter, one has to model the space in which the adversary can search for a perturbation using an axis-aligned ellipsoid in l 2 and an axis-aligned box in l ∞ .

Original: Using an (cid:15) -ball where the radius depends linearly on the individual parameter sets (Li et al., 2018; Cicek & Soatto, 2019; Wu et al., 2020) would either give the adversary too little or too much attack space.
Revised: Using an (cid:15) -ball where the radius depends linearly on the individual parameter sets (Li et al., 2018; Cicek & Soatto, 2019; Wu et al., 2020) would either give the adversary too little or too much attack space.

Original: Projecting onto an axis-aligned ellipsoid in l 2 corresponds to solving the following optimization problem (Gabay & Mercier, 1976; Dai, 2006), which does not have a closed-form solution:
Revised: Projecting onto an axis-aligned ellipsoid in l 2 corresponds to solving the following optimization problem (Gabay & Mercier, 1976; Dai, 2006), which does not have a closed-form solution:

Original: [Equation]
Revised: [Equation]

Original: where W = diag ( | Θ | (cid:12) ζ ) + I · ζ const , c = Θ and m =
Revised: where W = diag ( | Θ | (cid:12) ζ ) + I · ζ const , c = Θ and m =

Original: Θ ∗ + α (cid:12) v .
Revised: Θ ∗ + α (cid:12) v .

Original: Because of the computational overhead this would incur, we only consider the l ∞ case in our experiments.
Revised: Because of the computational overhead this would incur, we only consider the l ∞ case in our experiments.

Original:  R ESULTS
Revised:  R ESULTS

Original: The ultra-low power consumption of mixed-signal neuromorphic chips make them suitable for edgeapplications, such as always-on voice detection (Cho et al., 2019), vibration monitoring (Gies et al., [Figure]
Revised: The ultra-low power consumption of mixed-signal neuromorphic chips make them suitable for edgeapplications, such as always-on voice detection (Cho et al., 2019), vibration monitoring (Gies et al., 2021) or always-on face recognition (Liu et al., 2019).

Original: For this reason, we consider two compact network architectures in our experiments: A Spiking Recurrent Neural Network (SRNN) with roughly 65k trainable parameters and a conventional CNN with roughly 500k trainable parameters (see S1 for more information).
Revised: For this reason, we consider two compact network architectures in our experiments: A Long Short-term spiking recurrent Neural Network (LSNN) with roughly 65k trainable parameters and a conventional CNN with roughly 500k trainable parameters (see S1 for more information).

Original: We trained these models to perform three different tasks:
Revised: We trained these models to perform three different tasks:

Original: • Speech command detection of 6 classes (Warden, 2018); • ECG-anomaly detection on 4 classes (Bauer et al., 2019); and • Fashion-MNIST
Revised: • Speech command detection of 6 classes (Warden, 2018); • ECG-anomaly detection on 4 classes (Bauer et al., 2019); and • Fashion-MNIST

Original: (F-MNIST): clothing-image classiﬁcation on 10 classes (Xiao et al., 2017).
Revised: (F-MNIST): clothing-image classiﬁcation on 10 classes (Xiao et al., 2017).

Original: We compared several training and attack methods, beginning with a standard Stochastic Gradient
Revised: We compared several training and attack methods, beginning with a standard Stochastic Gradient

Original: Descent (SGD) approach using the Adam optimizer (Kingma & Ba, 2015) (“Standard”).
Revised: Descent (SGD) approach using the Adam optimizer (Kingma & Ba, 2015) (“Standard”).

Original: Learning rate varied by architecture, but was kept constant when comparing training methods on an architecture.
Revised: Learning rate varied by architecture, but was kept constant when comparing training methods on an architecture.

Original: We examined networks trained with dropout (Srivastava et al., 2014), AWP (Wu et al., 2020),  ABCD (Cicek & Soatto, 2019), and Entropy-SGD (Chaudhari et al., 2019).
Revised: We examined networks trained with dropout (Srivastava et al., 2014), AWP (Wu et al., 2020), AMP (Zheng et al., 2020), ABCD (Cicek & Soatto, 2019), and Entropy-SGD (Chaudhari et al., 2019).

Original: The adversarial perturbations used in AWP and ABCD were adapted to our mismatch model (i.e. magnitude-dependent in l ∞ ).
Revised: The adversarial perturbations used in AWP and ABCD were adapted to our mismatch model (i.e. magnitude-dependent in l ∞ ) unless stated otherwise.

Original: We further adapted AWP so that the input data is never attacked.
Revised: AMP was not adapted.

Original: Unless stated otherwise, a dropout probability of 0.3 was used in the dropout models; the regularization parameter β Rob in our method was set to 0.25 and γ in AWP was set to 0.1. 
Revised: A dropout probability of 0.3 was used in the dropout models and γ in AWP was set to 0.1. When

Original: When Gaussian noise was applied to the weights during the forward pass (Murray & Edwards, 1994) a  standard deviation of 0.3  was used.
Revised: Gaussian noise was applied to the weights during the forward pass (Murray & Edwards, 1994) a relative standard deviation of 0.3 times the weight magnitude was used ( η train = 0 . 3 ).

Original: For Entropy-SGD, we set the number of inner iterations to 10 with a Langevin learning rate of 0.1.
Revised: For EntropySGD, we set the number of inner iterations to 10 with a Langevin learning rate of 0.1.

Original: Because Entropy-SGD and ABCD have inner loops, the number of total epochs were reduced accordingly.
Revised: Because Entropy-SGD and ABCD have inner loops, the number of total epochs were reduced accordingly.

Original: All other models were trained for the same number of epochs (no early stopping) and the model with the highest validation accuracy was selected.
Revised: All other models were trained for the same number of epochs (no early stopping) and the model with the highest validation accuracy was selected.

Original: We trained 168 models in parallel on a CPU cluster, with each model requiring on average 12 hours of compute time on a single CPU with 16 cores and 32Gb RAM.

Original: − 1 .
Revised: [Figure]

Original:  5 − 1 . 0 − 0 . 5 α 0 .0 .0 .0 .1 .
Revised: 10 2 10 3 10 4 10 5 10 6 10 7 T inf (s) 91.

Original: C r o ss E n t r o p y L o ss F-MNIST CNN
Revised: T e s t a cc .

Original: AWPBetaForwardNoise+Beta DropoutForwardNoiseStandard − 1 .
Revised: ( % ) attack =0. FP Baseline rob =0.0, train =0.

Original:  5 − 1 . 0 − 0 .
Revised: 10 2 10 3 10 4 10 5 10 6 10 7 T inf (s) 91. T e s t a cc .

Original: [-α 0 . 
-]
Revised: [-α 0 . 
-]( % ) attack =0.

Original: C r o ss E n t r o p y L o ss ECG SRNN − 1 . 5 − 1 . 0 − 0 .
Revised: 10 2 10 3 10 4 10 5 10 6 10 7 T inf (s) 91. T e s t a cc .

Original: α 0 .0 .
Revised: ( % ) attack =0.

Original: C r o ss E n t r o p y L o ss
Revised: 10 2 10 3 10 4 10 5 10 6 10 7 T inf (s) 91. T e s t a cc .

Original: Speech SRNN
Revised: ( % ) attack =0. r o b

Original: Effectiveness of adversarial weight attack We examined the strength of our adversarial weight attack during inference and training.
Revised: Effectiveness of adversarial weight attack We examined the strength of our adversarial weight attack during inference and training.

Original: Standard networks trained using gradient descent alone with no additional regularization (Fig.
Revised: Standard networks trained using gradient descent alone with no additional regularization (Fig.

Original: 1a, “Standard”) were disrupted badly by our adversarial attack during inference ( ζ = 0 . 1 ; ﬁnal mean test accuracy 91 .
Revised: S5a, “Standard”) were disrupted badly by our adversarial attack during inference ( ζ = 0 . 1 ; ﬁnal mean test accuracy 91 .

Original: 40% → 17 .
Revised: 40% → 17 .

Original: , and this was not ameliorated by further training.
Revised: , and this was not ameliorated by further training.

Original: When our adversarial attack was implemented during training (Fig.
Revised: When our adversarial attack was implemented during training (Fig.

Original: 1a, β rob = 0 . 1 ), the trained network was protected from disruption both during training and during inference (ﬁnal test accuracy 91 .
Revised: S5a, β rob = 0 . 1 ), the trained network was protected from disruption both during training and during inference (ﬁnal test accuracy 91 .

Original: 97% → 78 .
Revised: 97% → 78 .

Original: Our adversarial attack degrades network performance signiﬁcantly more than a random perturbation.
Revised: Our adversarial attack degrades network performance signiﬁcantly more than a random perturbation.

Original: Because our adversary uses PGA during the attack, it approximates a worst-case perturbation of the network within an ellipsoid around the nominal weights Θ .
Revised: Because our adversary uses PGA during the attack, it approximates a worst-case perturbation of the network within an ellipsoid around the nominal weights Θ .

Original: We compared the effect of our attack against a random weight perturbation (random point on ζ − ellipsoid) of equal magnitude.
Revised: We compared the effect of our attack against a random weight perturbation (random point on ζ − ellipsoid) of equal magnitude.

Original: For increasing perturbation size during inference (Fig. 1c; ζ ), our adversarial attack disrupted the performance of the standard network signiﬁcantly more than a random perturbation (test acc.
Revised: For increasing perturbation size during inference (Fig. S5c; ζ ), our adversarial attack disrupted the performance of the standard network signiﬁcantly more than a random perturbation (test acc.

Original: 40% → 17 .
Revised: 40% → 17 .

Original: 50% (attack) vs. 91 .
Revised: 50% (attack) vs. 91 .

Original: 40% → 90 .
Revised: 40% → 90 .

Original: 63% (random) for ζ = 0 . 1 ).
Revised: 63% (random) for ζ = 0 . 1 ).

Original: When our adversarial attack was applied during training (Fig.
Revised: When our adversarial attack was applied during training (Fig.

Original: 1d; β rob = 0 . 1 ), the network was protected against both random and adversarial attacks for magnitudes up to ζ = 0 .
Revised: S5d; β rob = 0 . 1 ), the network was protected against both random and adversarial attacks for magnitudes up to ζ = 0 .

Original: 7 and ζ = 0 . 1 , respectively.
Revised: 7 and ζ = 0 . 1 , respectively.

Original: Flatness of the weight-loss landscape Under our deﬁnition of robustness, the network output should change only minimally when the network parameters are perturbed.
Revised: Flatness of the weight-loss landscape Under our deﬁnition of robustness, the network output should change only minimally when the network parameters are perturbed.

Original: This corresponds to a loss surface that is close to ﬂat in weight space.
Revised: This corresponds to a loss surface that is close to ﬂat in weight space.

Original: We measured the test weight-loss landscape for trained networks, compared over alternative training methods and for several architectures (Fig. 2).
Revised: We measured the test weight-loss landscape for trained networks, compared over alternative training methods and for several architectures (Fig. 1).

Original: We examined only cross-entropy loss(i.e. task loss) over the test set, and not the adversarial attack loss component (KL divergence loss; see Eq. 2).
Revised: We examined only cross-entropy loss over the test set, and not the adversarial attack loss component (KL divergence loss; see Eq. 2).

Original: For each trained network, we chose a random vector v ∼ N ( 0 , ζ | Θ | ) and calculated L Nat ( f ( X test , Θ + α · v ) , y test ) for many evenly-spaced α ∈ [ − 2 , 2] .
Revised: For each trained network, we chose a random vector v ∼ N ( 0 , ζ | Θ | ) and calculated L cce ( f ( X test , Θ + α · v ) , y test ) for many evenly-spaced α ∈ [ − 2 , 2] .

Original: This process was repeated 5 times for ζ = 0 . 2 , and the means plotted in Fig.
Revised: This process was repeated 5 times for ζ = 0 . 2 , and the means plotted in Fig.

Original: Weight-loss landscapes for the individual trials are shown in Fig. S2.
Revised: Weight-loss landscapes for the individual trials are shown in Fig. S2.

Original: Our adversarial training approach found minima of trained parameters Θ in ﬂatter areas of the weightloss landscape, compared with all other approaches examined (ﬂatter curves in Fig. 2).
Revised: Our adversarial training approach found minima of trained parameters Θ in ﬂatter areas of the weightloss landscape, compared with all other approaches examined (ﬂatter curves in Fig. 1).

Original: In most cases our training approach also found deeper minima at lower  L cce , reﬂecting better task performance.
Revised: In most cases our training approach also found deeper minima at lower categorical cross-entropy loss ( L cce ), reﬂecting better task performance.

Original: These results are reﬂected in the better generalization performance of our approach (see Table 1).
Revised: These results are reﬂected in the better generalization performance of our approach (see Table 1).

Original: Not surprisingly, dropout and AWP also lead to ﬂatter minima than the Standard network with no regularization.
Revised: Not surprisingly, dropout and AWP also lead to ﬂatter minima than the Standard network with no regularization.

Original: ABCD and Entropy-SGD were not included in Fig.
Revised: ABCD and Entropy-SGD were not included in Fig.because they did not outperform the Standard model. Improved robustness for PCM based CiM With the increasing interest in CiM devices for accelerated inference and energy-efﬁcient edge computing (Sebastian et al., 2020), the problem of deploying a model that is robust to noise originating from the device physics of PCM cells has gained in signiﬁcance. We investigated the effect of our training method on the robustness of networks that are simulated to run on PCM-based CiM hardware (for details on the simulator, see SM 5). [Figure]

Revised: inference

Original: 2 because they did not outperform  the Standard model.
Revised: Networks trained with our method outperform state-of-the-art networks deployed on PCM-based CiM. Currently, the method that has proven to yield the best performance on CiM hardware is training with noise on the network parameters during the forward pass. We adapted this method by adding our algorithm and show that we consistently outperform the conventional method (see Figure S14) for a wide range of hyperparameters, and even surpass the FP baseline (a model trained without noise injection and evaluated on a standard PC) for some conﬁgurations (see Figure 2). Following experiments conducted in (Joshi et al., 2020), we used Resnet32 (He et al., 2015) trained on Cifar (Krizhevsky, 2009).

Revised: Injecting Gaussian noise on the weights during the forward pass yields strong improvements compared to the standard network.

Revised: We show that by adding our method, we consistently improve this robustness by a signiﬁcant amount (see Figure 3).

Revised: We furthermore improve the scalability of our method by using a pretrained model and fewer steps for the adversary.

Revised: Our algorithm incurs an additional training time that scales linearly with the number of attack steps used in the adversary (note that we cache the necessary gradients for the Jacobian calculation).

Revised: To alleviate this additional time, we show that our method produces good results even for just one single adversarial step.

Revised: Figure S13 shows the resulting performance when varying the number of attack steps used by the adversary.

Revised: It should be noted that all results reported on PCM robustness were obtained using three adversarial steps and a pretrained model.

Original: Network robustness against parameter mismatch We evaluated the ability of our training method to protect against simulated device mismatch.
Revised: Network robustness against parameter mismatch We evaluated the ability of our training method to protect against simulated device mismatch.

Original: We introduced frozen parameter noise into models trained with adversarial attack, with noise modelled on that observed in neuromorphic processors (Joshi et al., 2020; Büchel et al., 2021).
Revised: We introduced frozen parameter noise into models trained with adversarial attack, with noise modelled on that observed in neuromorphic processors (Joshi et al., 2020; Büchel et al., 2021).

Original: In these devices, uncertainty associated with each weight parameter is approximately normally distributed around the nominal value, with a standard deviation that scales with the weight magnitude (Eq 1).
Revised: In these devices, uncertainty associated with each weight parameter is approximately normally distributed around the nominal value, with a standard deviation that scales with the weight magnitude (Eq 1).

Original: We measured test accuracy under simulated a
Revised: We measured test accuracy under simulated random mismatch for 100 samples across two model instances. A comparison of our method against standard training is shown in Fig. For mismatch levels up to 70% ( ζ = 0 . 7 ), our approach protected signiﬁcantly against simulated mismatch for all three tasks examined ( p < 2 × 10 − 8 in all cases; U test). A more detailed comparison between the different models is given in Table 1.

Revised: Network robustness against direct adversarial attack on task performance Our training approach improves network robustness against mismatch parameter noise.

Original: R o bu s t
Revised: We further evaluated the robustness of our trained networks against a parameter adversary that directly attacks task performance, by performing PGA on the cross-entropy loss L cce . Note that this is separate from the adversary used in our training method, which attacks the boundary loss (Eq.2). The AWP method uses the cross-entropy loss to ﬁnd adversarial parameters during training. Nevertheless, we found that our method consistently outperforms all other compared methods for increasing attack magnitude ζ (Fig. 5).

Revised: In networks trained with our method, the adversary needed to perform a considerably larger a

Original:  S t a nd a r d 0.1 0 .0 .0 .0 .0 .1 .
Revised: R o bu s t S t a nd a r d 0.1 0 .0 .0 .0 .0 .1 .

Original: T e s t a cc .
Revised: T e s t a cc .

Original: 0.3 Mismatch level ( ζ ) 0.
Revised: 0.3 Mismatch level ( ζ ) 0.

Original: StandardOurs
Revised: StandardOurs

Original: R o bu s t b
Revised: R o bu s t b

Original: G r o und t r u t h
Revised: G r o und t r u t h

Original: S t a nd a r d 0.1 0 .0 .0 .0 .0 .1 .0.
Revised: S t a nd a r d 0.1 0 .0 .0 .0 .0 .1 .0.

Original: 0.3 Mismatch level ( ζ ) 0.
Revised: 0.3 Mismatch level ( ζ ) 0.

Original: c
Revised: c

Original: R o bu s t
Revised: R o bu s t

Original: S t a nd a r d 0.1 0 .0 .0 .0 .0 .1 .0.
Revised: S t a nd a r d 0.1 0 .0 .0 .0 .0 .1 .0.

Original: 0.3 Mismatch level ( ζ ) 0.
Revised: 0.3 Mismatch level ( ζ ) 0.

Original: [Figure]
Revised: [Figure]

Original:  e s t
Revised: T e s t

Original: [Figure]

Original:  cc 
Revised: A cc .

Original: [Figure]
Revised: [Figure]

Original: random mismatch for 100 samples across two model instances. A comparison of our method against standard training is shown in Fig. For mismatch levels up to70% ( ζ = 0 . 7 ), our approach protected signiﬁcantly against simulated mismatch for all three tasks examined ( p < 2 × 10 − 8 in all cases; U test). A more detailed comparison between the different models is given in Table 1.
Revised: attack to signiﬁcantly reduce performance (test accuracy < 70%).

Original: Network robustness against direct adversarial attack on task performance Our training approach improves network robustness against mismatch parameter noise.

Original: We further evaluated the robustness of our trained networks against a parameter adversary that directly attacks task performance, by performing PGA on the cross-entropy loss L Nat .

Original: Note that this is separate from the adversary used in our training method, which attacks the boundary loss (Eq.2).

Original: The AWP method uses the cross-entropy loss to ﬁnd adversarial parameters during training.

Original: Nevertheless, we found that our method consistently outperforms all other compared methods for increasing attack magnitude ζ (Fig. 4).

Original: In networks trained with our method, the adversary needed to perform a considerably larger attack to signiﬁcantly reduce performance (test accuracy < 70%).

Original: ABCD and Entropy-SGD were not included in the comparison because they did not outperform the standard network.
Revised: ABCD and Entropy-SGD were not included in the comparison because they did not outperform the standard network.

Original: [Figure] Veriﬁable robustness for SRNNs We investigated the provable robustness of SRNNs trained using our method using abstract interpretation (Cousot & Cousot, 1977;
Revised: Veriﬁable robustness for LSNNs We investigated the provable robustness of LSNNs trained using our method using abstract interpretation (Cousot & Cousot, 1977;

Original: Gehr et al., 2018; Mirman et al., 2018).
Revised: Gehr et al., 2018; Mirman et al., 2018).

Original: In this analysis a function f ( x, Θ) (in our case, a neural network with input x and parameters Θ ) is overapproximated using an abstract domain .
Revised: In this analysis a function f ( x, Θ) (in our case, a neural network with input x and parameters Θ ) is overapproximated using an abstract domain .

Original: We specify the weights Θ in our network as an interval parameterised by the attack size ζ , spanning [Θ − ζ | Θ | , Θ+ ζ | Θ | ] .
Revised: We specify the weights Θ in our network as an interval parameterised by the attack size ζ , spanning [Θ − ζ | Θ | , Θ + ζ | Θ | ] .

Original: Computations through the network are performed on these intervals rather than on discrete values.

Original: By propagating the intervals through a network over a test set, we obtain output logits that are also expressed as intervals.

Original: The ﬁnal classiﬁcation of the network is made using an arg max operator.

Original: For provability of network performance, we consider that a test sample x is correctly classiﬁed when the lower bound of the logit interval for the correct class is the maximum lower bound across all logit intervals, and when the logit interval for the correct class is disjoint from the other logit intervals.

Original: When the logit interval for the correct class overlaps with another logit interval, that test sample is not considered to be provably correctly classiﬁed.

Original: We examined the proportion of provably correctly classiﬁed test samples for the Speech and ECG tasks, under a range of attack sizes ζ , and comparing our approach against standard gradient descent and against training with forward-pass noise only (Fig. 5).
Revised: We examined the proportion of provably correctly classiﬁed test samples for the Speech and ECG tasks, under a range of attack sizes ζ , and comparing our approach against standard gradient descent and against training with forward-pass noise only (Fig. S9).

Original: We found that our approach is provably more correct over increasing attack size ζ (higher veriﬁed test accuracy).
Revised: We found that our approach is provably more correct over increasing attack size ζ (higher veriﬁed test accuracy).

Original: Note that interval domain analysis provides a relatively loose bound (Gehr et al., 2018), with the implication that the results here probably underestimate the true performance of our method.
Revised: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We proposed a new training approach that includes adversarial attacks on the parameter space during training.
Revised: We proposed a new training approach that includes adversarial attacks on the parameter space during training.

Original: Our attack disrupts the behaviour of a network by seeking a small weight perturbation that causes a maximal change in network output, by iteratively applying PGA.

Original: During training we jointly optimize a natural- and a robustness loss, where the robustness loss captures the susceptibility of the network to the adversarial attack.

Original: We evaluated our method on three tasks (F-MNIST; Speechcommands and ECG analysis), and measured the robustness of our approach against several forms of weight-space perturbation during inference.

Original: We compared our approach against several other training methods that could be expected to provide robustness against parameter perturbations.

Original: Our proposed adversarial attack was signiﬁcantly stronger than random weight perturbations at disrupting the performance of a trained network.
Revised: Our proposed adversarial attack was signiﬁcantly stronger than random weight perturbations at disrupting the performance of a trained network.

Original: Including the adversarial attack during training signiﬁcantly protected the trained network from weight perturbations during inference.
Revised: Including the adversarial attack during training signiﬁcantly protected the trained network from weight perturbations during inference.

Original: Our approach found minima in the weight-loss landscape that usually corresponded to lower loss values, and were always in ﬂatter regions of the loss landscape.
Revised: Our approach found minima in the weight-loss landscape that usually corresponded to lower loss values, and were always in ﬂatter regions of the loss landscape.

Original: This indicates that our approach found network solutions that are less sensitive to parameter variation, and therefore more robust.
Revised: This indicates that our approach found network solutions that are less sensitive to parameter variation, and therefore more robust.

Original: Our approach was more robust than several other methods for inducing robustness and good generalisation.[Table] ForwardNoise [Table]
Revised: Our approach was more robust than several other methods for inducing robustness and good generalisation.

Original: Mismatch [Table] To the best of our knowledge, our work represents the ﬁrst example of interval bound propagation applied to SNNs, and the ﬁrst application of parameter-space adversarial attacks to promote network robustness against device mismatch for mixed-signal compute.
Revised: To the best of our knowledge, our work represents the ﬁrst example of interval bound propagation applied to SNNs, and the ﬁrst application of parameter-space adversarial attacks to promote network robustness against device mismatch for mixed-signal compute.

Original: Our experiments only considered the impact of weight perturbations, and did not examine the inﬂuence of uncertainty in other parameters of mixed-signal neuromorphic processors such as time constants or spiking thresholds.
Revised: Our experiments only considered the impact of weight perturbations, and did not examine the inﬂuence of uncertainty in other parameters of mixed-signal neuromorphic processors such as time constants or spiking thresholds.

Original: Our approach can be adapted to include adversarial attacks in the full network parameter space, increasing the robustness of spiking networks.
Revised: Our approach can be adapted to include adversarial attacks in the full network parameter space, increasing the robustness of spiking networks.

Original: The technique of interval bound propagation can also be applied to these additional network parameters.
Revised: The technique of interval bound propagation can also be applied to these additional network parameters.

Original: We did not quantize network parameters either during or after training, in this work.
Revised: We did not quantize network parameters either during or after training, in this work.

Original: This quantization step, ideally performed during training, is required to deploy networks to specialized neuromorphic inference hardware. It is not obvious how our adversarial attacks would interact with quantization during the training process. 
Revised: On some platforms (Moradi et al., 2018) it is necessary to deploy quantized weights and it is unclear how our adversarial attacks would interact with quantization during the training process. However, most PCM-based CiM hardware does not require quantization during training in order to get good performance (Joshi et al., 2020).

Original: Per-device training for a device with known calibrated parameter noise is likely to achieve the highest possible deployed performance on that single device.
Revised: Per-device training for a device with known calibrated parameter noise is likely to achieve the highest possible deployed performance on that single device.

Original: However, this approach has signiﬁcant drawbacks.
Revised: However, this approach has signiﬁcant drawbacks.

Original: Firstly, each device must be either measured / calibrated accurately — not a trivial requirement — or trained with the device in the forward inference pass of the training loop.
Revised: Firstly, each device must be either measured / calibrated accurately — not a trivial requirement — or trained with the device in the forward inference pass of the training loop.

Original: Secondly, training must be performed individually for each device, entailing signiﬁcant logistical problems if the training is conducted in the factory or inside a consumer product.
Revised: Secondly, training must be performed individually for each device, entailing signiﬁcant logistical problems if the training is conducted in the factory or inside a consumer product.

Original: Thirdly, this approach will retain full sensitivity to parameter variation on the device.
Revised: Thirdly, this approach will retain full sensitivity to parameter variation on the device.

Original: Parameter drift, which is a known issue for many memristive devices, would therefore degrade performance unless re-training is performed regularly on the deployed device.

Original: In contrast, our training approach provides a way to train once for deployment at scale, while being robust to both fabrication mismatch and parameter drift.

Original: Our method improves the performance of neural networks deployed to inference hardware that include computational non-idealities.
Revised: Our method improves the performance of neural networks deployed to inference hardware that include computational non-idealities.

Original: For example, NN processors with crossbar architectures based on novel memory devices such as RRAM and PCMmemristors (Verma et al., 2019) display uncertainty in stored memory values as well as value drift over time (Le Gallo et al., 2018; Wu et al., 2019;
Revised: For example, NN processors with crossbar architectures based on novel memory devices such as RRAM and PCM (Verma et al., 2019) display uncertainty in stored memory values as well as conductance drift over time (Le Gallo et al., 2018b; Wu et al., 2019;

Original: Joshi et al., 2020).
Revised: Joshi et al., 2020).

Original: Analog neurons and synapses in mixed-signal NN processors, for example SNN inference processors (Moradi et al., 2018), exhibit variation in weights and neuron parameters across a processor.
Revised: Analog neurons and synapses in mixed-signal NN processors, for example SNN inference processors (Moradi et al., 2018), exhibit variation in weights and neuron parameters across a processor.

Original: We showed that our training approach ﬁnds network solutions that are insensitive to mismatchinduced parameter variation.
Revised: We showed that our training approach ﬁnds network solutions that are insensitive to mismatch-induced parameter variation.

Original: Our networks can therefore be deployed to inference devices with computational non-idealities with only minimal reduction in task performance, and without requiring per-device calibration or model training.
Revised: Our networks can therefore be deployed to inference devices with computational non-idealities with only minimal reduction in task performance, and without requiring per-device calibration or model training.

Original: This reduction in per-device handling implies a considerable reduction in expense when deploying at commercial scale.
Revised: This reduction in per-device handling implies a considerable reduction in expense when deploying at commercial scale.

Original: Our method therefore brings low-power neuromorphic inference processors closer to commercial viability.
Revised: Our method therefore brings low -power neuromorphic inference processors closer to commercial viability.

Original: E THICS STATEMENTS
Revised: E THICS STATEMENTS

Original: The authors declare no conﬂicts of interest.
Revised: The authors declare no conﬂicts of interest.

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: All code required to re-train the networks and reproduce all experiments described in this work are provided as supplementary material.
Revised: All code required to re-train the networks and reproduce all experiments described in this work are provided as supplementary material.

Original: Pre-trained networks and all data pre-processing is performed by code in supplementary material
Revised: Pre-trained networks and all data pre-processing is performed by code in supplementary material

Original: S UPPLEMENTARY M ATERIAL S PIKING RNN A RCHITECTURE
Revised: S UPPLEMENTARY M ATERIAL S PIKING RNN A RCHITECTURE

Original: The dynamics of the spiking model (Bellec et al., 2018) can be summarised by a set of differential equations (Eq. 3).
Revised: The dynamics of the spiking model (Bellec et al., 2018) can be summarised by a set of differential equations (Eq. 3).

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: refractory
Revised: refractory

Original: [Equation]
Revised: [Equation]

Original: where ρ β = e − dt /τ ada and ρ V =
Revised: where ρ β = e − dt /τ ada and ρ V =

Original: e − dt /τ .
Revised: e − dt /τ .

Original: The variables B describe the spiking thresholds with spike-frequency adaptation.
Revised: The variables B describe the spiking thresholds with spike-frequency adaptation.

Original: The vector o t denotes the population spike train at time t .
Revised: The vector o t denotes the population spike train at time t .

Original: The membrane potentials V have a time constant τ and the adaptive threshold time constant is denoted by τ ada .
Revised: The membrane potentials V have a time constant τ and the adaptive threshold time constant is denoted by τ ada .

Original: The speech signals and ECG traces fed into the network are represented as currents I in .
Revised: The speech signals and ECG traces fed into the network are represented as currents I in .

Original: Since the derivative of the spiking function with respect to its input is mostly 0 we use a surrogate gradient that is explicitly deﬁned as
Revised: Since the derivative of the spiking function with respect to its input is mostly 0 we use a surrogate gradient that is explicitly deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where E is the error and d is the dampening factor.
Revised: where E is the error and d is the dampening factor.

Original: To get the ﬁnal prediction of the network, we average the population spike trains along the time-axis z avg and compute
Revised: To get the ﬁnal prediction of the network, we average the population spike trains along the time-axis z avg and compute

Original: [Equation]
Revised: [Equation]

Original: CNN A RCHITECTURE
Revised: CNN A RCHITECTURE

Original: Our architecture comprises two convolutional blocks (2 × [4 × 4, 64 channels, MaxPool, ReLU]), followed by three dense layers ( N = 1600 , 256 , 64 , ReLU) and a softmax layer.
Revised: Our architecture comprises two convolutional blocks (2 × [4 × 4, 64 channels, MaxPool, ReLU]), followed by three dense layers ( N = 1600 , 256 , 64 , ReLU) and a softmax layer.

Original: All weights and kernels are initialized using the Glorot normal initialisation (Glorot & Bengio, 2010).
Revised: All weights and kernels are initialized using the Glorot normal initialisation (Glorot & Bengio, 2010).

Original: Using this architecture, we achieved a
Revised: Using this architecture, we achieved a

Original: D ERIVATION OF J ACOBIAN
Revised: D ERIVATION OF J ACOBIAN

Original: Under the assumption that α = ζ  (cid:12)| Θ | N steps and p = ∞ , we can rewrite the inner loop of Algorithm 1 to begin[Equation] = [Equation]
Revised: Under the assumption that α = ζ attack (cid:12)| Θ | N steps and p = ∞ , we can rewrite the inner loop of Algorithm 1 to begin

Original: to

Original: [Equation]

Original: do

Original: [Equation]
Revised: [Equation]

Original: end end
Revised: end end

Original: By rewriting Θ ∗ in the form of Θ ∗ =
Revised: By rewriting Θ ∗ in the form of Θ ∗ =

Original: Θ
Revised: Θ

Original: + ∆Θ
Revised: + ∆Θ

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: From this the Jacobian can be easily calculated.
Revised: From this the Jacobian can be easily calculated.

Original: Plugging in the deﬁntion for α we get
Revised: Plugging in the deﬁntion for α we get

Original: [Equation]
Revised: [Equation]

Original: M ISMATCH MODEL
Revised: M ISMATCH MODEL

Original: To model the parameter noise introduced by component mismatch we used a Gaussian distribution where the mean is the nominal noise-free weight value and the standard deviation depends linearly on the weight value.
Revised: To model the parameter noise introduced by component mismatch we used a Gaussian distribution where the mean is the nominal noise-free weight value and the standard deviation depends linearly on the weight value.

Original: This model realistically captures the behaviour of parameter mismatch on a mixedsignal neuromorphic SNN inference processor (Moradi et al., 2018).
Revised: This model realistically captures the behaviour of parameter mismatch on a mixedsignal neuromorphic SNN inference processor (Moradi et al., 2018).

Original: Fig.
Revised: Fig.

Original: S1 shows the quantiﬁed parameter mismatch recorded directly from neuromorphic HW, over a range of nominal parameter values and for several neuronal and synaptic parameters.
Revised: S1 shows the quantiﬁed parameter mismatch recorded directly from neuromorphic HW, over a range of nominal parameter values and for several neuronal and synaptic parameters.

Original: The measured mismatch parameter variation follows an approximately Gaussian distribution where the standard deviation depends linearly on the mean.
Revised: The measured mismatch parameter variation follows an approximately Gaussian distribution where the standard deviation depends linearly on the mean.

Original: [Figure]
Revised: [Figure]

Original: W EIGHT LOSS - LANDSCAPE VISUALIZATION
Revised: W EIGHT LOSS - LANDSCAPE VISUALIZATION

Original: We characterized the shape of the weight loss-landscape by plotting the categorical cross entropy loss for varying levels of noise added to the weights of the trained network.
Revised: We characterized the shape of the weight loss-landscape by plotting the categorical cross entropy loss for varying levels of noise added to the weights of the trained network.

Original: In each trial, we picked a random vector v ∼ N ( 0 , ζ | Θ | ) and evaluated the categorical cross entropy of the whole test set given the weights Θ + α · v , where α ∈ [ − 2 , 2] and ζ = 0 .
Revised: In each trial, we picked a random vector v ∼ N ( 0 , ζ | Θ | ) and evaluated the categorical cross entropy of the whole test set given the weights Θ + α · v , where α ∈ [ − 2 , 2] and ζ = 0 .

Original: As we show in Figure S2, the variance of the individual 1D weight loss-landscapes is small.
Revised: As we show in Figure S2, the variance of the individual 1D weight loss-landscapes is small.

Original: Table S1 quantiﬁes the ﬂatness of the illustrated weight loss-landscapes.
Revised: Table S1 quantiﬁes the ﬂatness of the illustrated weight loss-landscapes.

Original: As can be seen, our method combined with adding noise during the forward pass yields the ﬂattest landscapes.
Revised: As can be seen, our method combined with adding noise during the forward pass yields the ﬂattest landscapes.

Original: [Table]
Revised: [Table]

Revised: t r o p y L o ss

Original: [Figure]
Revised: [Figure]

Original: n t

Original: [Figure]
Revised: [Figure]

Revised: C r o ss E n t r o p y L o ss

Original:  A TTACKING KL- DIVERGENCE LOSS DURING INFERENCE
Revised: Speech LSNN A TTACKING KL- DIVERGENCE LOSS DURING INFERENCE

Original: While the adversary in AWP attacks the task loss directly, i.e. max L cat. cross-entropy ( f (Θ ∗ , X ) , y ) , the adversary in our training algorithm attacks the KL divergence max KL ( f (Θ , X ) , f (Θ ∗ , X )) .
Revised: While the adversary in AWP attacks the task loss directly, i.e. max L cce ( f (Θ ∗ , X ) , y ) , the adversary in our training algorithm attacks the KL divergence max KL ( f (Θ , X ) , f (Θ ∗ , X )) .

Original: This implies that our parameter attack seeks simply to change the response of the network in any way, and is agnostic to the task itself.
Revised: This implies that our parameter attack seeks simply to change the response of the network in any way, and is agnostic to the task itself.

Original: In the main results we attack the cross-entropy task loss during inference, in order to not give an undue advantage to our training approach.
Revised: In the main results we attack the cross-entropy task loss during inference, in order to not give an undue advantage to our training approach.

Original: Here we show that our training approach also provides robustness against parameter attack on the cross-entropy loss during inference.
Revised: Here we show that our training approach also provides robustness against parameter attacks on the KL-divergence loss during inference.

Original: Fig.
Revised: Fig.

Original: S3 shows the adversarial robustness of the methods for an adversary that attacks the KLdivergence rather than the cross-entropy loss.
Revised: S3 shows the adversarial robustness of the methods for an adversary that attacks the KLdivergence rather than the cross-entropy loss.

Original: When the network is attacked by maximizing the
Revised: When the network is attacked by maximizing the

Original: KL-divergence between the normal and attacked network, our adversarially-trained networks are more robust than AWP, SGD and forward-noise.
Revised: KL-divergence between the normal and attacked network, our adversarially-trained networks are more robust than AWP, SGD and forward-noise.

Original: Because the parameter attack used here during inference as well as in the inner optimization loop of the training procedure is the same, this result is expected, and serves as a sanity check that our networks indeed learn to defend against the attack they were trained against.
Revised: Because the parameter attack used here during inference as well as in the inner optimization loop of the training procedure is the same, this result is expected, and serves as a sanity check that our networks indeed learn to defend against the attack they were trained against.

Original: [Figure]
Revised: [Figure]

Original: the network is attacked by maximizing the KL-divergence between the normal and attacked network, our adversarially trained networks are more robust than standard SGD or AWP.
Revised: network is attacked by maximizing the KL-divergence between the normal and attacked network, our adversarially trained networks are more robust than standard SGD or AWP.

Original: E FFECT OF CONSTANT VERSUS RELATIVE PARAMETER NOISE
Revised: E FFECT OF CONSTANT VERSUS RELATIVE PARAMETER NOISE

Original: As described in the main text, parameter noise that has constant magnitude (for example, Gaussian noise with ﬁxed standard deviation) is trivial to protect against by increasing weight magnitudes.
Revised: As described in the main text, parameter noise that has constant magnitude (for example, Gaussian noise with ﬁxed standard deviation) is trivial to protect against by increasing weight magnitudes.

Original: We examined this effect by training MLPs with an adversary that employs Gaussian noise with ﬁxed std.
Revised: We examined this effect by training MLPs with an adversary that employs Gaussian noise with ﬁxed std.

Original: dev.
Revised: dev.

Original: (cid:15) = 0 .
Revised: (cid:15) = 0 .

Original: Figure S4 (right) illustrates the test accuracy of two MLPs trained on F-MNIST over the course of training.
Revised: Figure S4 (right) illustrates the test accuracy of two MLPs trained on F-MNIST over the course of training.

Original: Using an inverse weight decay term, the weight-magnitude of one network is forced to increase to 2.0 over the course of training (black crosses, Θ ∗ ).
Revised: Using an inverse weight decay term, the weight-magnitude of one network is forced to increase to 2.0 over the course of training (black crosses, Θ ∗ ).

Original: The weight magnitude of the other network (red crosses) is limited during training to 0.2 (red crosses; Θ ).
Revised: The weight magnitude of the other network (red crosses) is limited during training to 0.2 (red crosses; Θ ).

Original: One can observe that as the weight magnitude of the increasing magnitude network Θ ∗ increases, also the robustness to
Revised: One can observe that as the weight magnitude of the increasing magnitude network Θ ∗ increases, also the robustness to

Original: Gaussian noise ( (cid:15) = 0 . 2 ) increases (blue), while the performance of the small magnitude network Θ remains poor (cyan). 
Revised: Gaussian noise ( (cid:15) = 0 . 2 ) increases (blue), while the performance of the small magnitude network Θ remains poor (cyan). [Figure]

Revised: E FFECT OF ADVERSARIAL REGULARIZATION DURING TRAINING

Original: [Figure]
Revised: [Figure]

Original: E FFECT OF VARYING β ROB
Revised: E FFECT OF VARYING β rob

Original: We additionally quantify the trade-off between test loss and robustness of our algorithm by repeating the experiment in Figure 2 at different values of β rob .
Revised: We additionally quantify the trade-off between test loss and robustness of our algorithm by repeating the experiment in Figure 1 at different values of β rob .

Original: As shown in Figure S5, increasing β rob ﬂattens the weight loss-landscape, effectively increasing the robustness of the model.
Revised: As shown in Figure S6, increasing β rob ﬂattens the weight loss-landscape, effectively increasing the robustness of the model.

Original: To further substantiate this claim, we repeated the experiment of Figure 1 with the same values of β rob .
Revised: To further substantiate this claim, we repeated the experiment of Figure S5 with the same values of β rob .

Original: As Figure S7 shows, increasing β rob and therefore increasing the ﬂatness of the landscape yields increased robustness to random, as well as, adversarial perturbations.
Revised: As Figure S8 shows, increasing β rob and therefore increasing the ﬂatness of the landscape yields increased robustness to random, as well as, adversarial perturbations.

Original: We note that, relative to the baseline, the test loss does not consistently increase with increasing values of β rob .
Revised: We note that, relative to the baseline, the test loss does not consistently increase with increasing values of β rob .

Original: We hypothesize that this is due to the increased generalization capability of our networks.
Revised: We hypothesize that this is due to the increased generalization capability of our networks.

Original: To check whether this is indeed the case, we repeated the experiment from Fig.
Revised: To check whether this is indeed the case, we repeated the experiment from Fig.

Original: S5 using the loss computed on the training set.
Revised: S6 using the loss computed on the training set.

Original: As Fig.
Revised: As Fig.

Original: S6 shows, increasing values of β rob lead to ﬂatter minima and higher loss on the training set, which is the exact trade-off to be expected from the formulation of our loss function.
Revised: S7 shows, increasing values of β rob lead to ﬂatter minima and higher loss on the training set, which is the exact trade-off to be expected from the formulation of our loss function.

Original: However, the increased generalization capability that follows from a ﬂatter loss-landscape seems to disrupt this trade-off.
Revised: However, the increased generalization capability that follows from a ﬂatter loss-landscape seems to disrupt this trade-off.

Original: As a result, when choosing β rob one should aim at choosing the highest value that still yields good performance on the validation set.
Revised: As a result, when choosing β rob one should aim at choosing the highest value that still yields good performance on the validation set.

Original: [Figure]
Revised: [Figure]

Revised: n t r o p y L o ss

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We additionally trained the CNN on the F-MNIST data for a range of β rob
Revised: We additionally trained the CNN on the F-MNIST data for a range of β rob

Original: [Figure]
Revised: [Figure]

Original: and measured the network robustness to attacks of varying magnitude ζ (Fig. S7).
Revised: and measured the network robustness to attacks of varying magnitude ζ (Fig. S8).

Original: We compared the effectiveness of random attack versus adversarial attack during inference, evaluated on the test set.
Revised: We compared the effectiveness of random attack versus adversarial attack during inference, evaluated on the test set.

Original: We found that increasing β rob improved the robustness of the network to both random and adversarial attack during inference.
Revised: We found that increasing β rob improved the robustness of the network to both random and adversarial attack during inference.

Original: [Figure]
Revised: [Figure]

Revised: V ERIFIABLE ROBUSTNESS FOR LSNN S

Revised: Computations through the network are performed on intervals rather than on discrete values.

Revised: By propagating the intervals through a network over a test set, we obtain output logits that are also expressed as intervals and can therefore determine whether a sample will always be classiﬁed correctly.

Revised: The ﬁnal classiﬁcation of the network is made using an arg max operator.

Revised: For provability of network performance, we consider that a test sample x is correctly classiﬁed when the lower bound of the logit interval for the correct class is the maximum lower bound across all logit intervals, and when the logit interval for the correct class is disjoint from the other logit intervals.

Revised: When the logit interval for the correct class overlaps with another logit interval, that test sample is not considered to be provably correctly classiﬁed.

Revised: Note that interval domain analysis provides a relatively loose bound (Gehr et al., 2018), with the implication that the results here probably underestimate the true performance of our method.

Revised: [Figure]

Original: W IDE - MARGIN NETWORK ACTIVATIONS
Revised: W IDE - MARGIN NETWORK ACTIVATIONS

Original: Murray et al.
Revised: Murray et al.

Original: show that adding random forward noise to the weights of a network during the forward pass implicitly adds a regularizer of the form Θ 2 i,j (cid:16) ∂o k,l ∂ Θ i,j (cid:17) 2 to the network weights (Murray & Edwards, 1994).
Revised: show that adding random forward noise to the weights of a network during the forward pass implicitly adds a regularizer of the form Θ 2 i,j (cid:16) ∂o k,l ∂ Θ i,j (cid:17) 2 to the network weights (Murray & Edwards, 1994).

Original: When sigmoid activation functions are used, this regularizer favors high or low activations.
Revised: When sigmoid activation functions are used, this regularizer favors high or low activations.

Original: When implementing interval bound propagation for SRNNs, intervals over spiking activity must be computed by passing intervals through the spiking threshold function.
Revised: When implementing interval bound propagation for LSNNs, intervals over spiking activity must be computed by passing intervals through the spiking threshold function.

Original: As a result, intervals for spiking activity become either [0 , 0] for neurons that never emit a spike regardless of weight attack; [1 , 1] for neurons that always emit a spike; and [0 , 1] for neurons for which activity becomes uncertain in the presence of weight attack.
Revised: As a result, intervals for spiking activity become either [0 , 0] for neurons that never emit a spike regardless of weight attack; [1 , 1] for neurons that always emit a spike; and [0 , 1] for neurons for which activity becomes uncertain in the presence of weight attack.

Original: By deﬁnition, a robust network should promote bounds [0 , 0] and [1 , 1] , where the activity of the network is unchanged by weight attack.
Revised: By deﬁnition, a robust network should promote bounds [0 , 0] and [1 , 1] , where the activity of the network is unchanged by weight attack.

Original: Robust network conﬁgurations should therefore avoid states where the membrane potentials of neurons are close to the ﬁring threshold.
Revised: Robust network conﬁgurations should therefore avoid states where the membrane potentials of neurons are close to the ﬁring threshold.

Original: To see whether this was also the case for our SRNNs, we investigated the distribution of the membrane potentials on a batch of test examples for a network that was trained with- and without noise during the forward pass.
Revised: To see whether this was also the case for our LSNNs, we investigated the distribution of the membrane potentials on a batch of test examples for a network that was trained with- and without noise during the forward pass.

Original: We found that robust networks exhibited a broader distribution of membrane potentials, with comparatively less distribution mass close to the ﬁring threshold (Fig. S8).
Revised: We found that robust networks exhibited a broader distribution of membrane potentials, with comparatively less distribution mass close to the ﬁring threshold (Fig. S10).

Original: This indicates that neurons in the robust network spend more time in a “safe” regime where a small change in the weights cannot trigger an unwanted spike, or remove a desired spike. 
Revised: This indicates that neurons in the robust network spend more time in a “safe” regime where a small change in the weights cannot trigger an unwanted spike, or remove a desired spike. [Figure] E FFECT OF VARYING (cid:15) PGA IN AWP

Revised: In addition to attacking the network parameters

Original: [Figure]
Revised: [Figure]

Revised: AWP (Wu et al., 2020) also attacks the input using

Revised: PGA.

Revised: Since the relation between robustness to weight- and input-space perturbations is still unclear

Revised: [Figure]

Revised: we performed additional sweeps over the attack size in the input space.

Revised: Figure S11 demonstrates that attacking the input during training generally does not improve the robustness, with the exception for the network trained on the speech dataset.

Revised: We also note that attacking the inputs improved robustness for larger mismatch values, but generally degraded performance for the small values

Revised: [Figure]

Revised: We swept various values of (cid:15) pga , the parameter that determines the maximum perturbation in l ∞ for the AWP algorithm.

Revised: T RAINING OF CNN USED FOR PCM- BASED C I M SIMULATION

Revised: The CNN that was used for this series of experiments is Resnet32 (He et al., 2015) trained on the

Revised: Cifar10 (Krizhevsky, 2009) dataset.

Revised: The CNN was generally trained for 300 epochs using a batch size of 256.

Revised: We used SGD with an initial learning rate of 0.001 that was decreased by a multiplicative factor of 0.2 after epochs 60, 120 and 160.

Revised: Additionally, Nesterov momentum (Nesterov, 1983) was used with a value of 0.9 and weight decay with a value of 5e-

Revised: [Figure]

Revised: two standard deviations during training to avoid outliers causing excessive amounts of noise following the model that relies on the maximum weight value.

Revised: V ARYING THE NUMBER OF ATTACK STEPS FOR THE PCM- BASED C I M SIMULATION

Revised: [Figure]

Revised: [Figure]

Revised: train = 90.

Revised: T e s t a cc .

Revised: [Figure]

Revised: [Figure]

Revised: P ERFORMANCE OF VARYING HYPERPARAMETERS FOR THE PCM- BASED C I M SIMULATION

Revised: In this experiment we show that the choice of hyperparameters is generally not very important to outperform the baseline that was trained with noise injection.

Revised: However, in order to surpass the FP baseline, i.e. the model that was trained without noise injection and evaluated on a standard PC, one has to tune the hyperparameters in order to obtain a combination that yields the highest performance.

Revised: Figure S14 illustrates this sweep.

Revised: Each row represents a different value of η train that was used for training the baseline model (red).

Revised: Each column represents a different attack size and the different hues of blue correspond to varying values of β rob .

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: r o b

Revised: Our method is resilient to variations in hyperparameters.

Revised: However, to obtain conﬁgurations where even the FP baseline is surpassed, one has to ﬁne-tune the method.

Revised: PCM NOISE MODEL

Revised: Analog CiM comes in various ﬂavors, depending on the memory technology used.

Revised: In this paper, we assume the use of PCM devices, which have been heavily studied in the context of analog CiM accelerators (Joshi et al., 2020; Nandakumar et al., 2019; Boybat et al., 2018).

Revised: PCM-based, or, more generally, Non-Volatile Memory (NVM)-based architectures, essentially perform Matrix-VectorMultiplications (MVMs) using Kirchhoff’s current law.

Revised: The weights of the matrix are organized as differential pairs in order to account for positive and negative weights.

Revised: When storing a neural network, each weight matrix is programmed into the NVM devices by applying short electrical pulses (Nandakumar et al., 2020b).

Revised: Because of various noise sources, this process is often imprecise and exhibits noise on the weights, termed "programming noise".

Revised: Additionally, PCM devices suffer from 1 /f and telegraph noise, adding even more noise during inference ("read noise").

Revised: At last,

Revised: PCM devices also drift due to the underlying physical properties (Le Gallo et al., 2018a).

Revised: Although the effect of drift can mostly be alleviated by scaling the output of the MVM (a method called

Revised: Global Drift Compensation (GDC)), the non-uniform drift of the devices still leads to performance degradation over time.

Revised: In the simulator that we used, we model these three main sources of noise, analog-to-digital and digital-to-analog converters, GDC and splitting of the MVM to account for smaller tile sizes (typically each crossbar is 256 × 256 ).

Revised: Initially, the clipped weights are mapped to target conductances in a differential manner, i.e. the weight matrix is split into two conductance matrices that both represent the positive and negative weights as conductances.

Revised: The target conductances typically range from zero to G max , where G max is assumed to be 25 µS .

Revised: After mapping the weights to the target conductances G T , the programming noise is simulated (these statistical models assume the conductances to be normalized): G P =

Revised: G T + N (0 , σ P ) where σ P = max ( − 1 .

Revised: 1731 G 2 T

Revised: + 1 .

Revised: 9650 G T + 0 .

Revised: After the conductances have been programmed they drift over time, with the conductance of a device typically following G D = G P ( t/t c ) − ν , where ν is the drift coefﬁcient, t is the time at inference, and t c is the time the conductances were programmed.

Revised: Additionally, the drift coefﬁcient is modelled to follow a Gaussian distribution.

Revised: This makes it typically hard to correct for drift and it is the main reason why drift is a problem in PCM based CiM devices.

Revised: Finally, the read noise is modelled using a Gaussian: G R ∼ N ( G D , σ nG ( t ) ) , where σ nG ( t ) =

Revised: G D ( t ) Q (cid:112) log (( t + t r ) /t r ) with Q = min (0 . 0088 /G

Revised: 65 T , 0 . 2) and t r = 250 ns and t is the time at inference.

Revised: For the experiments in this paper, we simulated the performance of the networks deployed on CiM hardware for up to one year.