Original: Exploring the Sharpened Cosine Similarity
Revised: Exploring the Sharpened Cosine Similarity

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Convolutional layers have long served as the primary workhorse for image classification.
Revised: Convolutional layers have long served as the primary workhorse for image classification.

Original: Recently, an alternative to convolution was proposed using the Sharpened
Revised: Recently, an alternative to convolution was proposed using the Sharpened

Original: Cosine Similarity (SCS), which in theory may serve as a better feature detector.
Revised: Cosine Similarity (SCS), which in theory may serve as a better feature detector.

Original: While multiple sources report promising results, there has not been to date a fullscale empirical analysis of neural network performance using these new layers.
Revised: While multiple sources report promising results, there has not been to date a fullscale empirical analysis of neural network performance using these new layers.

Original: Inour work, we explore SCS’s parameter behavior and potential as a drop-in replacement for convolutions in multiple CNN architectures benchmarked on CIFAR-10.
Revised: In our work, we explore SCS’s parameter behavior and potential as a drop-in replacement for convolutions in multiple CNN architectures benchmarked on CIFAR-10.

Original: We find that while SCS may not yield significant increases in accuracy, it may learnmore interpretable representations.
Revised: We find that while SCS may not yield significant increases in accuracy, it may learn more interpretable representations.

Original: We also find that, in some circumstances, SCSmay confer a slight increase in adversarial robustness.
Revised: We also find that, in some circumstances, SCS may confer a slight increase in adversarial robustness.

Original:  Introduction
Revised:  Introduction

Original: For decades, convolutional layers have served as the workhorses in neural network architecturesfor image classification.
Revised: For decades, convolutional layers have served as the workhorses in neural network architectures for image classification.

Original: Mathematically, a convolutional layer slides across an input image andcomputes the dot product between a signal s and a kernel k (see Rohrer [2022a]).
Revised: Mathematically, a convolutional layer slides across an input image and computes the dot product between a signal s and a kernel k (see [1]).

Original: However, initialexploration and discussion by Rohrer [2020] suggest that convolutional layers, while excellent imagefilters, may not be very good feature detectors.
Revised: However, initial exploration and discussion by [2] suggest that convolutional layers, while excellent image filters, may not be very good feature detectors.

Original: In addition, Luo et al. [2018] argue that because thedot product operation is unbounded, neural networks using convolutional layers may be vulnerable toincreased model variance, over-sensitivity, and a lack of generalizability.As such, Luo et al.
Revised: In addition, [3] argue that because the dot product operation is unbounded, neural networks using convolutional layers may be vulnerable to increased model variance, over-sensitivity and a lack of generalizability.

Original: [2018]explored replacing traditional convolutional layers with cosine similarity.
Revised: As such, [3] explored replacing traditional convolutional layers with cosine similarity.

Original: Mathematically, the cosinesimilarity (“CosSim") between s and k is defined as follows.
Revised: Mathematically, the cosine similarity (“CosSim") between s and k is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: Importantly, cosine similarity is bounded between − 1 and 1 , potentially reducing the varianceconcerns of standard CNNs.
Revised: Importantly, cosine similarity is bounded between − 1 and 1 , potentially reducing the variance concerns of standard CNNs.

Original: Traditionally, cosine similarity has been used extensively in textanalysis to quantify the “similarity" between two documents (see Singhal et al. [2001]), among othermachine learning tasks (see Tan et al.
Revised: Traditionally, cosine similarity has been used extensively in text-analysis to quantify the “similarity" between two documents (see [4]), among other machine learning tasks (see [5]).

Original: For deep-learning networks, Luo et al. [2018] foundthat incorporating cosine similarity in VGG-type CNNs outperformed standard convolution-basedalternatives using batch, weight, and layer normalization on CIFAR-10/100 and SVHN.
Revised: For deep-learning networks, [3] found that incorporating cosine similarity in VGG-type CNNs outperformed standard convolution-based alternatives using batch, weight, and layer normalization on CIFAR-10/100 and SVHN.

Original: Building on cosine similarity, Rohrer [2022a] proposed that one can “sharpen" the standard cosinesimilarity to better discern whether two vectors are similar by raising the standard cosine similarity toa power p , while preserving the sign of the original cosine value.
Revised: Building on cosine similarity, [1] proposed that one can “sharpen" the standard cosine similarity to better discern whether two vectors are similar by raising the standard cosine similarity to a power p , while preserving the sign of the original cosine value.

Original: To prevent numerical instabilityarising from the signal s potentially having a near-zero magnitude, Rohrer [2022a] presents thefollowing modified mathematical formulation for Sharpened Cosine Similarity (SCS), with a smallpositive scalar q added to || s || :
Revised: To prevent numerical instability arising from the signal s potentially having a near-zero magnitude, [1] presents the following modified mathematical formulation for Sharpened Cosine Similarity (SCS), with a small positive scalar q added to || s || :

Original: [Equation]
Revised: [Equation]

Original: In February 2020, Rohrer [2020] used a one-dimensional input vector to visually demonstrate that an SCS kernel was a more accurate feature detector than a standard convolution kernel.
Revised: In February 2020, [2] used a one-dimensional input vector to visually demonstrate that an SCS kernel was a more accurate feature detector than a standard convolution kernel.

Original: Specifically, Rohrer showed that the SCS kernel more consistently output large values in the presence of a desiredfeature and smaller values in the absence of such a feature, than a standard convolution kernel.
Revised: Specifically, Rohrer showed that the SCS kernel more consistently output large values in the presence of a desired feature and smaller values in the absence of such a feature, than a standard convolution kernel.

Original: Themath and examples provided by Rohrer [2020] are convincing that standard convolutions are betterfilters than detectors, and the hypothesis that such a change from convolutions to SCS would improvepredictive accuracy is intuitive and logical.
Revised: The math and examples provided by [2] are convincing that standard convolutions are better filters than detectors, and the hypothesis that such a change from convolutions to SCS would improve predictive accuracy is intuitive and logical.

Original: In this study, we more thoroughly investigate the SCS, exploring accuracy, efficiency, interpretability,and adversarial robustness as hypotheses of what may be improved by SCS.
Revised: In this study, we more thoroughly investigate the SCS, exploring accuracy, efficiency, interpretability, and adversarial robustness as hypotheses of what may be improved by SCS.

Original: Our experiments arerepeated using multiple different current network architectures to see if any benefits of SCS improvewith smaller network size, as hypothesized by current small-scale results of Rohrer.
Revised: Our experiments are repeated using multiple different current network architectures to see if any benefits of SCS improve with smaller network size, as hypothesized by current small-scale results of Rohrer.

Original: Our results findsome measurable differences between SCS and convolutional networks, but said differences do notyet rise to a level that would meaningfully impact current practitioners.
Revised: Our results find some measurable differences between SCS and convolutional networks, but said differences do not yet rise to a level that would meaningfully impact current practitioners.

Original:  Review of Related Work
Revised:  Review of Related Work

Original: Since the Rohrer [2020] Twitter thread, many have experimented with various implementations of the SCS layer.
Revised: Since the [2] Twitter thread, many have experimented with various implementations of the SCS layer.

Original: Initial results suggest that SCS is very parameter-efficient, which may be desirable in usecases with limited compute or wattage capacities.
Revised: Initial results suggest that SCS is very parameter-efficient, which may be desirable in use-cases with limited compute or wattage capacities.

Original: On MNIST, Pisoni [2022] reported achieving 99%accuracy using an SCS-based architecture with less than 1.4K parameters.
Revised: On MNIST, [6] reported achieving 99% accuracy using an SCS-based architecture with less than 1.4K parameters.

Original: Pisoni [2022] also observedthat SCS networks tended to produce more interpretable weights, work well with unscaled inputs,and require no normalization.
Revised: [6] also observed that SCS networks tended to produce more interpretable weights, work well with unscaled inputs, and require no normalization.

Original: It was also observed that SCS kernels could extract not only exactmatches of desired features, but also their opposites (i.e., with opposite sign).
Revised: It was also observed that SCS kernels could extract not only exact matches of desired features, but also their opposites (i.e., with opposite sign).

Original: As such, Pisoni [2022]recommended that SCS be paired with MaxAbsPool2d (taking the maximum of the absolute values ofan input), in contrast to the traditional MaxPool2d, for optimal performance.
Revised: As such, [6] recommended that SCS be paired with MaxAbsPool2d (taking the maximum of the absolute values of an input), in contrast to the traditional MaxPool2d, for optimal performance.

Original: On CIFAR-10, Rohrerpublished an SCS-based model in his scs-gallery (see Rohrer [2022b]) that claimed to achieve80% accuracy with only 25.2K parameters, an order of magnitude less than existing architectureswith comparable accuracy.
Revised: On CIFAR-10, Rohrer published an SCS-based model in his scs-gallery (see [7]) that claimed to achieve 80% accuracy with only 25.2K parameters, an order of magnitude less than existing architectures with comparable accuracy.

Original: Nestler [2022] experimented with scaling up SCS networks to largerdatasets and leveraging TPU computations, but found that SCS was computationally significantlyslower to train than convolutional layers.
Revised: [8] experimented with scaling up SCS networks to larger datasets and leveraging TPU computations, but found that SCS was computationally significantly slower to train than convolutional layers.

Original: This is primarily because the exponentiation to the p thpower in SCS does not parallelize well to GPUs and TPUs.
Revised: This is primarily because the exponentiation to the p th power in SCS does not parallelize well to GPUs and TPUs.

Original: In addition to the properties discussed above, members of the machine learning community havefound that the inclusion of SCS layers does not appear to produce more accurate models thanexisting architectures (see Rohrer [2022a]).
Revised: In addition to the properties discussed above, members of the machine learning community have found that the inclusion of SCS layers does not appear to produce more accurate models than existing architectures (see [1]).

Original: Others have also found that SCS architectures do notrequire nonlinear activations (e.g., ReLU, sigmoid), dropout layers, nor normalization layers (e.g., 
Revised: Others have also found that SCS architectures do not require nonlinear activations (e.g., ReLU, sigmoid), dropout layers, nor normalization layers (e.g., BatchNorm2d) after

Original: BatchNorm2d) after SCS layers.
Revised: SCS layers.

Original: Some researchers have also tried incorporating SCS into existingarchitectures such as ResNet (see Batchelor [2022]), compact transformers (see Walton [2022]),models for ASL classification (see Wagner [2022]), and GANs (see Zimonitrome [2022]), withlimited or mixed reported results.
Revised: Some researchers have also tried incorporating SCS into existing architectures such as ResNet (see [9]), compact transformers (see [10]), models for ASL classification (see [11]), and GANs (see [12]), with limited or mixed reported results.

Original: However, to the best of our knowledge, there do not exist anyformally published manuscripts on SCS experiments and behavior, especially in comparison totraditional convolutional layers.
Revised: However, to the best of our knowledge, there do not exist any formally published manuscripts on SCS experiments and behavior, especially in comparison to traditional convolutional layers.

Original:  Materials and Methods3.1 Architectures
Revised:  Materials and Methods 3.1 Architectures

Original: For initial exploratory experiments, we selected two network architectures to serve as our test platforms: first, an SCS network with 100K parameters from Rohrer’s scs-gallery , which we will call“Rohrer100K" (see Rohrer [2022b]); second, the ResNet18 network from PyTorch’s torchvisionlibrary with 11.6 million parameters.
Revised: For initial exploratory experiments, we selected two network architectures to serve as our test platforms: first, an SCS network with 100K parameters from Rohrer’s scs-gallery , which we will call “Rohrer100K" (see [7]); second, the ResNet18 network from PyTorch’s torchvision library with 11.6 million parameters.

Original: A ResNet18/20 architecture specially designed for CIFAR-10was also briefly used when comparing the weight behaviors of SCS layers versus their convolutionalcounterparts (see Idelbayev).
Revised: A ResNet18/20 architecture specially designed for CIFAR-10 was also briefly used when comparing the weight behaviors of SCS layers versus their convolutional counterparts (see [13]).

Original: For follow-on experiments, we selected four more originally SCS-basedarchitectures from Rohrer’s scs-gallery that we will call Rohrer25K, Rohrer47K, Rohrer68K, and
Revised: For follow-on experiments, we selected four more originally SCS-based architectures from Rohrer’s scs-gallery that we will call Rohrer25K, Rohrer47K, Rohrer68K, and

Original: Rohrer583K (named after their parameter counts, see Rohrer [2022b]).
Revised: Rohrer583K (named after their parameter counts, see [7]).

Original: The first two of these werereported to achieve 80%+ accuracy on CIFAR-10, while the last two were reported to achieve 90%+accuracy on standard CIFAR-10.
Revised: The first two of these were reported to achieve 80%+ accuracy on CIFAR-10, while the last two were reported to achieve 90%+ accuracy on standard CIFAR-10.

Original: These models were selected to explore SCS behavior across a widerange of parameter-counts.
Revised: These models were selected to explore SCS behavior across a wide range of parameter-counts.

Original: For the above architectures, our benchmark was the standard 32 × [-
 CIFAR-10 dataset.-]
Revised: For the above architectures, our benchmark was the standard 32 × [-
 CIFAR-10 dataset.-]32 CIFARdataset.

Original: To explore the behavior of SCS on higher-resolution images, we also explored 
Revised: To explore the behavior of SCS on higher-resolution images, we also explored SCS-based

Original: SCS-based ResNet18 variants on CIFAR-10 resized to 224 × 224 .
Revised: ResNet18 variants on CIFAR-10 resized to 224 × 224 .

Original: For each starting architecture, we tested every variant combination of the following settings: convolutional layer vs. SCS; MaxPool2d vs. MaxAbsPool2d; ReLU vs. no activation.
Revised: For each starting architecture, we tested every variant combination of the following settings: convolutional layer vs. SCS; MaxPool2d vs. MaxAbsPool2d; ReLU vs. no activation.

Original: For our follow-upexperiments on ResNet18 ( 224 × 224 ), in addition to the settings specified above, we also exploredvariants with or without BatchNorm2d.
Revised: For our follow-up experiments on ResNet18 ( 224 × 224 ), in addition to the settings specified above, we also explored variants with or without BatchNorm2d.

Original: When replacing one feature with another (e.g., convolutionalwith SCS), every attempt was made to carryover as many settings as possible from the originalstarting architecture (e.g., kernel size, number of filters, etc.).
Revised: When replacing one feature with another (e.g., convolutional with SCS), every attempt was made to carryover as many settings as possible from the original starting architecture (e.g., kernel size, number of filters, etc.).

Original: In addition, for each architecture family,all model variants using that architecture were trained with the same initial weights.
Revised: In addition, for each architecture family, all model variants using that architecture were trained with the same initial weights.

Original: We hope thatsuch practices allow us to more accurately assess the direct effects of transplanting SCS into existingarchitectures.
Revised: We hope that such practices allow us to more accurately assess the direct effects of transplanting SCS into existing architectures.

Original: Please see Appendix A for additional experimental details.
Revised: Please see Appendix A for additional experimental details.

Original: For all model variants, train and test accuracies, losses, and compute times (in seconds) were recordedper epoch.
Revised: For all model variants, train and test accuracies, losses, and compute times (in seconds) were recorded per epoch.

Original: The norms of the weights and gradients in each layer were also recorded per epoch forall variants.
Revised: The norms of the weights and gradients in each layer were also recorded per epoch for all variants.

Original: Vanilla gradient saliency maps were generated for each model variant to probe modelinterpretability.
Revised: Vanilla gradient saliency maps were generated for each model variant to probe model interpretability.

Original: In addition, projected gradient descent (PGD) adversarial attacks were performedagainst all model variants to further assess model interpretability.
Revised: In addition, projected gradient descent (PGD) adversarial attacks were performed against all model variants to further assess model interpretability.

Original: For variants using SCS layers, wealso recorded the p exponentiation parameter values of each SCS layer per epoch.
Revised: For variants using SCS layers, we also recorded the p exponentiation parameter values of each SCS layer per epoch.

Original:  Initial Exploratory Experiments on Rohrer100K and ResNet18 
Revised:  Initial Exploratory Experiments on Rohrer100K and ResNet

Original: We begin by exploring the accuracy and training efficiency, the nature and parameter behavior, andthe interpretability and adversarial robustness of SCS when used in the Rohrer100K and ResNet18architectures on 32 × 32 CIFAR-10.
Revised: We begin by exploring the accuracy and training efficiency, the nature and parameter behavior, and the interpretability and adversarial robustness of SCS when used in the Rohrer100K and ResNet architectures on 32 × 32 CIFAR-10.

Original: Initial testing suggested that SCS networks tended to require moreepochs of training to achieve their maximum test accuracy.
Revised: Initial testing suggested that SCS networks tended to require more epochs of training to achieve their maximum test accuracy.

Original: As such, all models using architecturesoriginally from scs-gallery were trained for 800 epochs, while all other models were trainedfor 200 epochs.
Revised: As such, all models using architectures originally from scs-gallery were trained for 800 epochs, while all other models were trained for 200 epochs.

Original: Please see subsection A.1 for more experimental setting details.
Revised: Please see subsection A.1 for more experimental setting details.

Original: Additional initialexploratory experiments regarding SCS parameter behavior can be found in Appendix C.
Revised: Additional initial exploratory experiments regarding SCS parameter behavior can be found in Appendix C.

Original: 4.1 Accuracy and Efficiency
Revised: 4.1 Accuracy and Efficiency

Revised: model variants on 32 × 32 CIFAR-10.

Revised: See Appendix

Revised: B for ResNet18 results.

Original: [Table]
Revised: [Table]

Original: From Table 1 above and Table 2 (see Appendix B)
Revised: From Table 1 above and Table 2 (see Appendix B)

Original: [Table]
Revised: [Table]

Original: we see that within the same architecture family, theuse of SCS layers does not yield a significant increase in accuracy (approx.
Revised: we see that within the same architecture family [Table] the use of SCS layers does not yield a significant increase in accuracy (approx.

Original: 78% for Rohrer100K and82% for ResNet18) on CIFAR-10.
Revised: 78% for Rohrer100K and 82% for ResNet18) on CIFAR-10.

Original: However, we do confirm that SCS is able to achieve comparableaccuracy performance to standard convolutional layers without the inclusion of ReLU nonlinearactivation functions, while convolutional layers must be paired with such nonlinear activations formaximum performance.
Revised: However [Table] we do confirm that SCS is able to achieve comparable accuracy performance to standard convolutional layers without the inclusion of ReLU nonlinear activation functions, while convolutional layers must be paired with such nonlinear activations for maximum performance.

Original: Yet, it appears that the inclusion of ReLU activations in SCS networks maystill provide a small increase in accuracy performance.
Revised: Yet, it appears that the inclusion of ReLU activations in SCS networks may still provide a small increase in accuracy performance.

Original: Interestingly, in Rohrer100K, we also findthat standard convolutional layers paired with MaxAbsPool2d (and ReLU) incur a significant dip inaccuracy performance.
Revised: Interestingly, in Rohrer100K, we also find that standard convolutional layers paired with MaxAbsPool2d (and ReLU) incur a significant dip in accuracy performance.

Original: But, this dip does not seem to occur with ResNet18.
Revised: But, this dip does not seem to occur with ResNet18.

Original: We found that SCS-based networks require significantly more time (seconds per epoch) to train thantheir corresponding convolution-based counterparts.
Revised: We found that SCS-based networks require significantly more time (seconds per epoch) to train than their corresponding convolution-based counterparts.

Original: This corroborates best practices and observationsthat others have found.
Revised: This corroborates best practices and observations that others have found.

Original: However, while all SCS-based ResNet18 variants were slower in evaluationtime than their convolution-based counterparts, all SCS-based Rohrer100K variants were noticeablyfaster in evaluation time than their convolution-based counterparts.
Revised: However, while all SCS-based ResNet18 variants were slower in evaluation time than their convolution-based counterparts, all SCS-based Rohrer100K variants were noticeably faster in evaluation time than their convolution-based counterparts.

Original: This suggests that SCS propertiesmay not unilaterally hold across all architectures.
Revised: This suggests that SCS properties may not unilaterally hold across all architectures.

Original: 4.2 Cosine Similarity (Unsharpened)
Revised: 4.2 Cosine Similarity (Unsharpened)

Original: We hypothesize that SCS does not require nonlinear activations like ReLU to achieve peak performance because the exponentiation by p (sharpening) in SCS renders the classification functionnonlinear.
Revised: We hypothesize that SCS does not require nonlinear activations like ReLU to achieve peak performance because the exponentiation by p (sharpening) in SCS renders the classification function nonlinear.

Original: In other words, we postulate that SCS could be interpreted as a combination of outputnormalization (via cosine similarity) and an exponential activation function.
Revised: In other words, we postulate that SCS could be interpreted as a combination of output normalization (via cosine similarity) and an exponential activation function.

Original: To test this hypothesis,we fix p = 1 in the SCS layers of Rohrer100K to test whether (unsharpened) cosine similarity on itsown would require an activation function.
Revised: To test this hypothesis, we fix p = 1 in the SCS layers of Rohrer100K to test whether (unsharpened) cosine similarity on its own would require an activation function.

Original: The maximum accuracy of cosine similarity coupled with
Revised: The maximum accuracy of cosine similarity coupled with

Original: ReLU was 76 .
Revised: ReLU was 76 .

Original: 04% , while the maximum accuracy of cosine similarity without ReLU was 56 .
Revised: 04% , while the maximum accuracy of cosine similarity without ReLU was 56 .

Original: This corroborates our hypothesis that SCS can be interpreted as a combination of normalization andan exponential activation function.
Revised: This corroborates our hypothesis that SCS can be interpreted as a combination of normalization and an exponential activation function.

Original: 4.3 Weight Norms of SCS Layers
Revised: 4.3 Weight Norms of SCS Layers

Original: To explore how the normalization and exponentiation components of SCS would affect the weightsused in its convolution component (versus a standard convolutional layer), we compare the weightsof a fully-trained convolution-based network versus those of a fully-trained SCS-based network.
Revised: To explore how the normalization and exponentiation components of SCS would affect the weights used in its convolution component (versus a standard convolutional layer), we compare the weights of a fully-trained convolution-based network versus those of a fully-trained SCS-based network.

Original: Toensure that the weights of our fully-trained convolution-based network represented the maximumperformance (reasonably) possible, we decided to use a ResNet18 architecture specifically-optimizedfor CIFAR-10 by Idelbayev (see Idelbayev), as opposed to the torchvision variant optimized for
Revised: To ensure that the weights of our fully-trained convolution-based network represented the maximum performance (reasonably) possible, we decided to use a ResNet18 architecture specifically-optimized for CIFAR-10 by Idelbayev (see [13]), as opposed to the torchvision variant optimized for

Original: ImageNet.
Revised: ImageNet.

Original: We trained two SCS-based variants, each only replacing the convolutional layers for SCS,and not altering any other model components.
Revised: We trained two SCS-based variants, each only replacing the convolutional layers for SCS, and not altering any other model components.

Original: However, one variant was trained using randomlyinitialized starting weights, while the other was trained using pre-trained weights specifically designedfor convolution-layers on CIFAR-10.
Revised: However, one variant was trained using randomlyinitialized starting weights, while the other was trained using pre-trained weights specifically designed for convolution-layers on CIFAR-10.

Original: We recorded the L2 norms of each SCS layer’s convolutionweights over each mini-batch.
Revised: We recorded the L2 norms of each SCS layer’s convolution weights over each mini-batch.

Original: From Figure 2 in Appendix B, we see that for the variant trained from scratch, the L2 norms ofeach SCS layer ’s weights did not appear to noticeably change over epoch time.
Revised: From Figure 2 in Appendix B, we see that for the variant trained from scratch, the L2 norms of each SCS layer’s weights did not appear to noticeably change over epoch time.

Original: However, for thevariant trained from pre-trained weights originally optimized for standard convolution layers, the L2norms of the SCS layer weights significantly monotonically increased over mini-batches.
Revised: However, for the variant trained from pre-trained weights originally optimized for standard convolution layers, the L norms of the SCS layer weights significantly monotonically increased over mini-batches.

Original: As such,these results suggest that SCS-based architectures prefer larger weights than their convolution-basedcounterparts.
Revised: As such, these results suggest that SCS-based architectures prefer larger weights than their convolution-based counterparts.

Original: The L2 norms reported at mini-batch 0 in the pre-trained plot are precisely the normsfound in the fully-trained convolution-based network.
Revised: The L2 norms reported at mini-batch 0 in the pre-trained plot are precisely the norms found in the fully-trained convolution-based network.

Original: We hypothesize that these observations can beexplained by SCS trying to learn a “sharper" representation.
Revised: We hypothesize that these observations can be explained by SCS trying to learn a “sharper" representation.

Original: Additionally, SCS may tolerate largerweights because of the output normalization.
Revised: Additionally, SCS may tolerate larger weights because of the output normalization.

Original: Perhaps larger weights may also contribute to moresensitive feature detection and differentiation.
Revised: Perhaps larger weights may also contribute to more sensitive feature detection and differentiation.

Original: 4.4 Saliency Maps
Revised: 4.4 Saliency Maps

Original: To investigate Pisoni’s observations regarding SCS interpretability, vanilla gradient saliency mapswere generated for convolution and SCS variants of Rohrer100K and ResNet18, presented as Figures3 and 4 in Appendix B.
Revised: To investigate Pisoni’s observations regarding SCS interpretability, vanilla gradient saliency maps were generated for convolution and SCS variants of Rohrer100K and ResNet18, presented as Figures 3 and 4 in Appendix B.

Original: In Figure 1 below, we provide one representative example of the saliencymaps of a convolution-based and an SCS-based Rohrer100K corresponding to an airplane (class0).
Revised: In Figure 1 below, we provide one representative example of the saliency maps of a convolution-based and an SCS-based Rohrer100K corresponding to an airplane (class 0).

Original: We observe that the convolution-based variant yields a “scattered" saliency map, while the SCSvariant yields a more focused map that corresponds to the triangular-shaped aircraft.
Revised: We observe that the convolution-based variant yields a “scattered" saliency map, while the SCS variant yields a more focused map that corresponds to the triangular-shaped aircraft.

Original: Overall, across
Revised: Overall, across

Original: Rohrer100K and ResNet18, our initial results suggest that SCS-based model variants tend to learnrepresentations that are more interpretable and focus on more critical parts of the image than theirconvolution-based counterparts.
Revised: Rohrer100K and ResNet18, our initial results suggest that SCS-based model variants tend to learn representations that are more interpretable and focus on more critical parts of the image than their convolution-based counterparts.

Original: 4.5 Robustness to Adversarial Attacks
Revised: 4.5 Robustness to Adversarial Attacks

Original: To further explore the interpretability of SCS-based networks, we performed projected gradientdescent (PGD) adversarial attacks on all Rohrer100K and ResNet18 model variants, varying attackstrength from ϵ = 0 .
Revised: To further explore the interpretability of SCS-based networks, we performed projected gradient descent (PGD) adversarial attacks on all Rohrer100K and ResNet18 model variants, varying attack strength from ϵ = 0 .

Original: 001 to ϵ = 0 .
Revised: 001 to ϵ = 0 .

Original: We posit that architectures with more interpretable representations (i.e., focus more on true features and signals, and less noise) should be more robust toadversarial attacks.
Revised: We posit that architectures with more interpretable representations (i.e., focus more on true features and signals, and less noise) should be more robust to adversarial attacks.

Original: From Figures 3 and 4 in Appendix B, we find that SCS-based Rohrer100K and
Revised: From Figures 3 and 4 in Appendix B, we find that SCS-based Rohrer100K and

Original: ResNet18 variants are noticeably more adversarially robust than their convolution-based counterparts,in terms of experiencing a slower decay in accuracy with increasing attack strength.
Revised: ResNet18 variants are noticeably more adversarially robust than their convolution-based counterparts, in terms of experiencing a slower decay in accuracy with increasing attack strength.

Original: The one outlierin the Rohrer100K trials was the convolution-based variant with MaxAbsPool2d and ReLU, whichwas also an outlier in terms of achieving only 64.96% accuracy (while other variants achieved high
Revised: The one outlier in the Rohrer100K trials was the convolution-based variant with MaxAbsPool2d and ReLU, which was also an outlier in terms of achieving only 64.96% accuracy (while other variants achieved high

Original: [Figure]
Revised: [Figure]

Original: 70%s or 80%s).
Revised: 70%s or 80%s).

Original: Notably, for ResNet18, the best SCS-based ResNet18 variant (incidentally, usingthe best practices suggested by Rohrer and the ML community) was particularly more adversariallyrobust than all convolution-based variants.
Revised: Notably, for ResNet18, the best SCS-based ResNet18 variant (incidentally, using the best practices suggested by Rohrer and the ML community) was particularly more adversarially robust than all convolution-based variants.

Original: As such, the initial adversarial experimental evidencefurther suggests that the incorporation of SCS layers may produce more interpretable networks.
Revised: As such, the initial adversarial experimental evidence further suggests that the incorporation of SCS layers may produce more interpretable networks.

Original:  Follow-up Experiments
Revised:  Follow-up Experiments

Original: To investigate the generalizability of our initial SCS findings across different-sized architectures, werepeated our previous experiments on four additional models from the scs-gallery : Rohrer25K,
Revised: To investigate the generalizability of our initial SCS findings across different-sized architectures, we repeated our previous experiments on four additional models from the scs-gallery : Rohrer25K,

Original: Rohrer47K, Rohrer68K, Rohrer100K (with new randomized initial weights), and Rohrer583K. Tofurther discern which properties of SCS can be attributed to normalization and which to exponentiation, we include a third potential layer option: an ablation of convolutions and SCS that we will call
Revised: Rohrer47K, Rohrer68K, Rohrer100K (with new randomized initial weights), and Rohrer583K. To further discern which properties of SCS can be attributed to normalization and which to exponentiation, we include a third potential layer option: an ablation of convolutions and SCS that we will call

Original: SharpenedSDP.
Revised: SharpenedSDP.

Original: SharpenedSDP stands for "sharpened strided dot product," and is mathematicallyequivalent to a standard convolutional layer raised to a learned p exponent, without the normalization.
Revised: SharpenedSDP stands for "sharpened strided dot product," and is mathematically equivalent to a standard convolutional layer raised to a learned p exponent, without the normalization.

Original: From our initial testing, we recalled that using convolutional layers without ReLU would almostalways lead to poor performance.
Revised: From our initial testing, we recalled that using convolutional layers without ReLU would almost always lead to poor performance.

Original: As such, we no longer tested this combination in the followingexperiments.
Revised: As such, we no longer tested this combination in the following experiments.

Original: To investigate SCS on higher-resolution images, we also repeated these experimentson ResNet18 with CIFAR-10 resized to 224 × 224 .
Revised: To investigate SCS on higher-resolution images, we also repeated these experiments on ResNet18 with CIFAR-10 resized to 224 × 224 .

Original: In addition to the settings described above, weincluded variants with/without BatchNorm2d (present in the original Resnet18 architecture).
Revised: In addition to the settings described above, we included variants with/without BatchNorm2d (present in the original Resnet18 architecture).

Original: We alsotried SharpenedSDP with 224 × 224 ResNet18, but all of these variants encountered vanishing and/orexploding gradients, so we omit them from discussion.
Revised: We also tried SharpenedSDP with 224 × 224 ResNet18, but all of these variants encountered vanishing and/or exploding gradients, so we omit them from discussion.

Original: 5.1 Accuracy and Efficiency
Revised: 5.1 Accuracy and Efficiency

Original: From Tables 3-7 in Appendix D, we find that our initial conclusions regarding SCS accuracy andefficiency generally hold strong across Rohrer25K through Rohrer583K.
Revised: From Tables 3-7 in Appendix D, we find that our initial conclusions regarding SCS accuracy and efficiency generally hold strong across Rohrer25K through Rohrer583K.

Original: We also observe that the
Revised: We also observe that the

Original: SharpenedSDP variants do not require activation functions, further corroborating our hypothesis thatthe exponentiation to the p th power is a substitute for nonlinear activation functions.
Revised: SharpenedSDP variants do not require activation functions, further corroborating our hypothesis that the exponentiation to the p th power is a substitute for nonlinear activation functions.

Original: In addition,we found that SCS and SharpenedSDP variants appear to consistently perform faster than theirconvolution-based counterparts on evaluation time.
Revised: In addition, we found that SCS and SharpenedSDP variants appear to consistently perform faster than their convolution-based counterparts on evaluation time.

Original: This is potentially one major advantage of
Revised: This is potentially one major advantage of

Original: SCS: while SCS-based networks might be slower to train, they are faster during evaluation.
Revised: SCS: while SCS-based networks might be slower to train, they are faster during evaluation.

Original: Finally,across Rohrer 25K through Rohrer583K, we found that model variants containing both ReLUand MaxAbsPool2d will almost always perform poorly, regardless of whether we are using SCS,
Revised: Finally, across Rohrer 25K through Rohrer583K, we found that model variants containing both ReLU and MaxAbsPool2d will almost always perform poorly, regardless of whether we are using SCS,

Original: SharpenedSDP, or standard convolutional layers.
Revised: SharpenedSDP, or standard convolutional layers.

Original: From Table 8 in Appendix D, we find that our initialconclusions regarding SCS accuracy and efficiency also generally hold strong on 224 × 224 ResNet18.
Revised: From Table 8 in Appendix D, we find that our initial conclusions regarding SCS accuracy and efficiency also generally hold strong on 224 × 224 ResNet18.

Original: However, in contrast to Rohrer25K through Rohrer583K, we do not see a dip in performanceassociated with the inclusion of both MaxAbsPool2d and ReLU.
Revised: However, in contrast to Rohrer25K through Rohrer583K, we do not see a dip in performance associated with the inclusion of both MaxAbsPool2d and ReLU.

Original: We also observed that the inclusionof BatchNorm2d, holding everything else constant, appears to grant a 3-4% increase in accuracy.
Revised: We also observed that the inclusion of BatchNorm2d, holding everything else constant, appears to grant a 3-4% increase in accuracy.

Original: 5.2 Saliency Maps and Interpretability
Revised: 5.2 Saliency Maps and Interpretability

Original: From the saliency maps for Rohrer25K through 583K, as shown in Figures 9-13 in Appendix D, we seethat our initial findings regarding SCS-based model variants learning sparser and more interpretablerepresentations appear to hold strong.
Revised: From the saliency maps for Rohrer25K through 583K, as shown in Figures 9-13 in Appendix D, we see that our initial findings regarding SCS-based model variants learning sparser and more interpretable representations appear to hold strong.

Original: In addition, we find significant evidence corroborating the hypothesis that the p exponentiation component in SCS is what creates such “sparse" saliencymaps.
Revised: In addition, we find significant evidence corroborating the hypothesis that the p exponentiation component in SCS is what creates such “sparse" saliency maps.

Original: We conclude as such because the SharpenedSDP saliency maps much more consistentlyresembled the SCS saliency maps than the convolution ones.
Revised: We conclude as such because the SharpenedSDP saliency maps much more consistently resembled the SCS saliency maps than the convolution ones.

Original: From Figure 14 in Appendix D, wefind that our conclusions regarding SCS versus convolutional variants’ saliency maps also extend tohigher-resolution 224 × 224 images.
Revised: From Figure 14 in Appendix D, we find that our conclusions regarding SCS versus convolutional variants’ saliency maps also extend to higher-resolution 224 × 224 images.

Original: 5.2.1 Adversarial Robustness
Revised: 5.2.1 Adversarial Robustness

Original: Repeating the PGD attack simulations on Rohrer25K - 583K, we found our results differing significantly from our initial Rohrer100K results.
Revised: Repeating the PGD attack simulations on Rohrer25K - 583K, we found our results differing significantly from our initial Rohrer100K results.

Original: This was perplexing, because the underlying constructionsof all the scs-gallery models were all very similar.
Revised: This was perplexing, because the underlying constructions of all the scs-gallery models were all very similar.

Original: From Figure 15 in Appendix D, we find that the
Revised: From Figure 15 in Appendix D, we find that the

Original: SharpenedSDP/MaxAbsPool2d/ReLU and convolution/MaxAbsPool2d/ReLU variants consistentlyoutperformed the top SCS-based variant.
Revised: SharpenedSDP/MaxAbsPool2d/ReLU and convolution/MaxAbsPool2d/ReLU variants consistently outperformed the top SCS-based variant.

Original: However, these two variants also consistently performed significantly lower in terms of accuracy, across all four scs-gallery architectures (sometimes only hitting 10% accuracy).
Revised: However, these two variants also consistently performed significantly lower in terms of accuracy, across all four scs-gallery architectures (sometimes only hitting 10% accuracy).

Original: To verify our observations, we simulated PGD attacks against Rohrer100K again,this time using a different set of randomized initial weights (see Figure 16 in Appendix D).
Revised: To verify our observations, we simulated PGD attacks against Rohrer100K again, this time using a different set of randomized initial weights (see Figure 16 in Appendix D).

Original: Again, wefound that SharpenedSDP/MaxAbsPool2d/ReLU and convolution/MaxAbsPool2d/ReLU were the topadversarially-robust variants (though their original accuracies were dismal).
Revised: Again, we found that SharpenedSDP/MaxAbsPool2d/ReLU and convolution/MaxAbsPool2d/ReLU were the top adversarially-robust variants (though their original accuracies were dismal).

Original: There did not seem to bea significant difference between the remaining SCS, SharpenedSDP, and convolution-based variants,but the top performers with decent accuracy appear to be SharpenedSDP/MaxAbsPool2d/noReLU,followed by SCS/MaxAbsPool2d/ReLU.
Revised: There did not seem to be a significant difference between the remaining SCS, SharpenedSDP, and convolution-based variants, but the top performers with decent accuracy appear to be SharpenedSDP/MaxAbsPool2d/noReLU, followed by SCS/MaxAbsPool2d/ReLU.

Original: PGD attacks on 224 × 224 ResNet18 variants also present a different narrative than the initial PGDattacks on the 32 × 32 ResNet18 variants.
Revised: PGD attacks on 224 × 224 ResNet18 variants also present a different narrative than the initial PGD attacks on the 32 × 32 ResNet18 variants.

Original: As shown in Figure 17 in Appendix D, for variantswithout BatchNorm2d, convolution-based variants were noticeably more robust to adversarial attack.
Revised: As shown in Figure 17 in Appendix D, for variants without BatchNorm2d, convolution-based variants were noticeably more robust to adversarial attack.

Original: However, with BatchNorm2d, the best SCS-based variant maintains a small but consistent advantageover the convolution-based variants.
Revised: However, with BatchNorm2d, the best SCS-based variant maintains a small but consistent advantage over the convolution-based variants.

Original: The differences are not nearly as pronounced as in the 32 ×[-
-]
Revised: The differences are not nearly as pronounced as in the 32 ×[-
-]

Original: CIFAR-10 experiments, however.
Revised: CIFAR-10 experiments, however.

Original: Upon closer inspection, it appears that the inclusion or exclusionof BatchNorm2d simply does not affect SCS-based variants’ adversarial robustness, but does significantly affect convolution-based variants’ robustness.
Revised: Upon closer inspection, it appears that the inclusion or exclusion of BatchNorm2d simply does not affect SCS-based variants’ adversarial robustness, but does significantly affect convolution-based variants’ robustness.

Original: The causes of this behavior are not immediatelyobvious, and further exploration is necessary.
Revised: The causes of this behavior are not immediately obvious, and further exploration is necessary.

Original:  Discussion and Future Work
Revised:  Discussion and Future Work

Original: In this manuscript, we find strong evidence supporting the following five generalizations about SCSperformance and behavior.
Revised: In this manuscript, we find strong evidence supporting the following five generalizations about SCS performance and behavior.

Original: First, SCS does not yield significantly higher accuracy than standardconvolution based layers.
Revised: First, SCS does not yield significantly higher accuracy than standard convolution based layers.

Original: Second, while SCS-based models are almost always slower to train(seconds per epoch) than their convolution-based counterparts, they frequently benefit from slightlyfaster evaluation times (seconds per epoch).
Revised: Second, while SCS-based models are almost always slower to train (seconds per epoch) than their convolution-based counterparts, they frequently benefit from slightly faster evaluation times (seconds per epoch).

Original: Third, SCS can be interpreted as a combination ofconvolution, normalization, and nonlinear exponential activation.
Revised: Third, SCS can be interpreted as a combination of convolution, normalization, and nonlinear exponential activation.

Original: As such, they are capable ofperforming excellently without additional normalization or nonlinear activations like ReLU.
Revised: As such, they are capable of performing excellently without additional normalization or nonlinear activations like ReLU.

Original: Fourth,
Revised: Fourth,

Original: SCS tends to yield more interpretable learned representations than convolutions.
Revised: SCS tends to yield more interpretable learned representations than convolutions.

Original: Fifth, in certainenvironments, SCS also yields potentially more adversarially robust networks.
Revised: Fifth, in certain environments, SCS also yields potentially more adversarially robust networks.

Original: However, the above properties were not absolute.
Revised: However, the above properties were not absolute.

Original: This suggests that characterizing SCS performancemay be more analytically and computationally-involved than solely swapping out convolutionallayers and associated components for SCS layers.
Revised: This suggests that characterizing SCS performance may be more analytically and computationally-involved than solely swapping out convolutional layers and associated components for SCS layers.

Original: It appears that SCS performance and properties canvary noticeably, depending on the existing architecture and other components present in the system.
Revised: It appears that SCS performance and properties can vary noticeably, depending on the existing architecture and other components present in the system.

Original: Significant future work will be required to establish accompanying best practices for SCS networksand potentially construct novel architectures that best leverage SCS’s unique properties.
Revised: Significant future work will be required to establish accompanying best practices for SCS networks and potentially construct novel architectures that best leverage SCS’s unique properties.

Original: For example,creating a ResNet bottleneck module optimized for SCS may not be as straightforward as swappingout convolutional layers for SCS and replacing MaxPool2d with MaxAbsPool2d.
Revised: For example, creating a ResNet bottleneck module optimized for SCS may not be as straightforward as swapping out convolutional layers for SCS and replacing MaxPool2d with MaxAbsPool2d.

Original: Given (unsharpened) cosine similarity’s established uses in text modeling, a natural applied extensionis to transplant SCS into existing text modeling architectures.
Revised: Given (unsharpened) cosine similarity’s established uses in text modeling, a natural applied extension is to transplant SCS into existing text modeling architectures.

Original: SCS’s interpretability properties alsowarrant its transplanting to malware classification models that currently use standard convolutions.
Revised: SCS’s interpretability properties also warrant its transplanting to malware classification models that currently use standard convolutions.

Original: Another direction is to explore the use of SCS in convolution-based transformer models.
Revised: Another direction is to explore the use of SCS in convolution-based transformer models.

Original: We concludethat SCS is a promising alternative to convolutional layers, with its own set of unique advantages anddrawbacks.
Revised: We conclude that SCS is a promising alternative to convolutional layers, with its own set of unique advantages and drawbacks.

Original: A Additional Experimental Details
Revised: A Additional Experimental Details

Original: To standardize model training, unless otherwise specified, we used an Adam optimizer, coupledwith a maximum learning rate of 0 .
Revised: To standardize model training, unless otherwise specified, we used an Adam optimizer, coupled with a maximum learning rate of 0 .

Original: 01 and a OneCycleLR learning rate scheduler (see Smith and
Revised: 01 and a OneCycleLR learning rate scheduler (see [14]).

Original: Topin [2017]).

Original: RandomCrop and RandomHorizontalFlip data augmentations were also applied to thetraining set.
Revised: RandomCrop and RandomHorizontalFlip data augmentations were also applied to the training set.

Original: Such settings were directly borrowed from training scripts in Rohrer’s scs-gallery .
Revised: Such settings were directly borrowed from training scripts in Rohrer’s scs-gallery .

Original: Initial experiments on Rohrer100K suggested that weight decay did not noticeably affect SCS-basedarchitectures’ performance, so for all results presented, weight decay was set to 0 .
Revised: Initial experiments on Rohrer100K suggested that weight decay did not noticeably affect SCS-based architectures’ performance, so for all results presented, weight decay was set to 0 .

Original: For reproducibilityand to rule out the effects of different weight initializations on model performance, we ensured thatall variants of the same base architecture (e.g., ResNet18 on 32 × 32 ) were trained with the same setof randomly-initialized weights.
Revised: For reproducibility [15–17] and to rule out the effects of different weight initializations on model performance, we ensured that all variants of the same base architecture (e.g., ResNet18 on 32 × 32 ) were trained with the same set of randomly-initialized weights.

Original: All experiments were performed using PyTorch 1.11/1.12 on Linux.
Revised: All experiments were performed using PyTorch 1.11/1.12 on Linux.

Original: The hardware available were8 NVIDIA Tesla P100 GPUs with 16 GB of RAM each, and 8 NVIDIA Tesla V100 GPUs with32 GB of RAM each.
Revised: The hardware available were 8 NVIDIA Tesla P100 GPUs with 16 GB of RAM each, and 8 NVIDIA Tesla V100 GPUs with 32 GB of RAM each.

Original: We gratefully acknowledge the use of the Sharpened Cosine Similarity and
Revised: We gratefully acknowledge the use of the Sharpened Cosine Similarity and

Original: MaxAbsPool2d PyTorch implementations borrowed from Rohrer’s public repository (see Rohrer[2022a]).
Revised: MaxAbsPool2d PyTorch implementations borrowed from Rohrer’s public repository (see [1]).

Original: A.1 Additional Details on Initial Exploratory Experiments
Revised: A.1 Additional Details on Initial Exploratory Experiments

Original: We only tested combinations of convolution or SCS, ReLU or no activation, and MaxPool2d or
Revised: We only tested combinations of convolution or SCS, ReLU or no activation, and MaxPool2d or

Original: MaxAbsPool2d.
Revised: MaxAbsPool2d.

Original: We retained the BatchNorm2d and AdaptiveAvgPool layers in all tested ResNet18variants in this section.
Revised: We retained the BatchNorm2d and AdaptiveAvgPool layers in all tested ResNet variants in this section.

Original: The accuracy results for ResNet18 in section 4 were obtained using a torchvision ResNet18originally designed for ImageNet, but we replaced the final fully-connected layer to account forthe 10 classes in CIFAR-10.
Revised: The accuracy results for ResNet18 in section 4 were obtained using a torchvision ResNet originally designed for ImageNet, but we replaced the final fully-connected layer to account for the 10 classes in CIFAR-10.

Original: During initial experimentation, we also tried transfer learning with
Revised: During initial experimentation, we also tried transfer learning with

Original: SCS networks by starting ResNet18 training using pre-trained ResNet18 weights for ImageNet.
Revised: SCS networks by starting ResNet18 training using pre-trained ResNet18 weights for ImageNet.

Original: Wefound that there was no significant difference between model variants trained from scratch or usingpre-trained weights.
Revised: We found that there was no significant difference between model variants trained from scratch or using pre-trained weights.

Original: As such, we only report results using randomly-initialized starting weights.
Revised: As such, we only report results using randomly-initialized starting weights.

Original: [Table]

Original: B Tables and Additional Figures from Initial Exploratory Experiments
Revised: B Tables and Additional Figures from Initial Exploratory Experiments

Revised: of initial ResNet18 model variants on 32 × 32 CIFAR-

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: B.2 Saliency Maps and Interpretability
Revised: B.2 Saliency Maps and Interpretability

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: B.3 Adversarial Robustness
Revised: B.3 Adversarial Robustness

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: c

Original: [Figure]
Revised: [Figure]

Original: PGD Attacks on ResNet18 Networks: (0.001,0.030)
Revised: PGD Attacks on ResNet18 Networks: (0.001,0.030)

Original: SCSConv2dno_activation reluMaxAbsPool2dMaxPool2d
Revised: SCSConv2dno_activation reluMaxAbsPool2dMaxPool2d

Original: C Additional Initial Experiments 
Revised: C Additional Initial Experiments C.1 P-values of SCS Layers

Original: C.1 P-values of SCS Layers

Original: We also explored the behavior of the p values in a SCS layer over epoch time.
Revised: We also explored the behavior of the p values in a SCS layer over epoch time.

Original: Using a fixed learningrate of 1 . 

Revised: Using a fixed learning rate of 1 .

Original: × 10 − 3 , coupled with a weight decay of 1 . 

Revised: × 10 − 3 , coupled with a weight decay of 1 .

Original: × 10 − 5 , we plotted the learned values of p forthe kernels in the first SCS layer of Rohrer100K (with MaxAbsPool2d, without ReLU).
Revised: × 10 − 5 , we plotted the learned values of p for the kernels in the first SCS layer of Rohrer100K (with MaxAbsPool2d, without ReLU).

Original: From Figure7 below, we find that there are not many dynamics of note.
Revised: From Figure 7 below, we find that there are not many dynamics of note.

Original: Monotonicity does not appear to hold.
Revised: Monotonicity does not appear to hold.

Original: C.2 Optuna Hyperparameter Tuning of Fixed- p Hyperparameter
Revised: C.2 Optuna Hyperparameter Tuning of Fixed- p Hyperparameter

Original: We also wished to investigate whether certain values of p were more amenable to SCS performancethan others.
Revised: We also wished to investigate whether certain values of p were more amenable to SCS performance than others.

Original: As a proxy, we used a custom SCS layer with p fixed to a certain specified value.
Revised: As a proxy, we used a custom SCS layer with p fixed to a certain specified value.

Original: Weswapped out the original layers in both Rohrer100K and torchvision ’s ResNet18 for these fixed- p
Revised: We swapped out the original layers in both Rohrer100K and torchvision ’s ResNet18 for these fixed- p

Original: SCS layers, and preserved all other existing components.
Revised: SCS layers, and preserved all other existing components.

Original: For simplicity, we forced all SCS layersin each network to share the same initialized and fixed value of p .
Revised: For simplicity, we forced all SCS layers in each network to share the same initialized and fixed value of p .

Original: We used the Optuna Bayesianhyperparameter tuning utility to find the most optimal values of p (see Akiba et al.
Revised: We used the Optuna Bayesian hyperparameter tuning utility to find the most optimal values of p (see [18]).

Original: Sincemultiple initial values of p needed to be tested, we restricted each Rohrer100K trial to 600 epochs(versus the standard 800 epochs) and each ResNet18 trial to 100 epochs (versus the standard 200epochs).
Revised: Since multiple initial values of p needed to be tested, we restricted each Rohrer100K trial to 600 epochs (versus the standard 800 epochs) and each ResNet18 trial to 100 epochs (versus the standard 200 epochs).

Original: We also enabled pruning to terminate non-promising p -value trials early.
Revised: We also enabled pruning to terminate non-promising p -value trials early.

Original: [Figure]
Revised: [Figure]

Original: Optuna results across both Rohrer100K and ResNet18 suggest that all tested values of p above ornear 1 . 0 appear to yield good performance.
Revised: Optuna results across both Rohrer100K and ResNet18 suggest that all tested values of p above or near 1 . 0 appear to yield good performance.

Original: This suggests that the specific value of p may not bethe most important – rather, it is the mere presence of the (increasing) exponentiation operation thatenables successful SCS performance.
Revised: This suggests that the specific value of p may not be the most important – rather, it is the mere presence of the (increasing) exponentiation operation that enables successful SCS performance.

Original: D Tables and Figures From Follow-up Experiments 
Revised: D Tables and Figures From Follow-up Experiments D.1 Accuracy and Efficiency

Original: D.1 Accuracy and Efficiency

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D.2 Saliency Maps and Interpretability
Revised: D.2 Saliency Maps and Interpretability

Original: In this section, we only display saliency maps for model variants that achieved an overall accuracy of60% or greater.
Revised: In this section [Table] we only display saliency maps for model variants that achieved an overall accuracy of 60% or greater.

Original: Models achieving lower accuracy than 60% may not be representative of their layers’overall behaviors
Revised: Models achieving lower accuracy than 60% may not be representative of their layers’ overall behaviors.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: serves to emphasize the signal.

Original: [Figure]
Revised: [Figure]

Original: All convolutional andSCS variants did not appear to capture any meaningful, human-interpretable featuresin the Such aresult (and many similar results encountered) suggest that SharpenedSDP itself may be a promising area of future exploration.
Revised: All convolutional and variants did not appear to capture any meaningful, human-interpretable features a many similar results encountered) suggest that SharpenedSDP itself may be a promising area of future exploration.

Original: Across examples, it appears that SharpenedSDP coupled with no activation tends to be a very promising combination.
Revised: Across examples, it appears that SharpenedSDP coupled with no activation tends to be a very promising combination

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Observe how the 4 convolution-based variants do not seem to produce human-interpretable saliency maps for the B-2 stealth bomber.
Revised: Observe how the 4 convolution-based variants do not seem to produce human-interpretable saliency maps for the B-2 stealth bomber.

Original: However, all 8 SCS variants produced saliency maps that were extremely similar to each other, all capturing the jagged bottom edge of the aircraft.
Revised: However, all 8 SCS variants produced saliency maps that were extremely similar to each other, all capturing the jagged bottom edge of the aircraft.

Original: The SCS-based variants’ saliency maps were also significantly sparser than that of the convolution-based variants.
Revised: The SCS-based variants’ saliency maps were also significantly sparser than that of the convolution-based variants.

Original: D.3 Adversarial Robustness
Revised: D.3 Adversarial Robustness

Original: [Figure]
Revised: [Figure]

Original: scs-gallery architecture variants on CIFAR-10 ( 32 × 32 )

Original: [Figure] [Figure]
Revised: [Figure] [Figure]