Original: Faster Attention Is What You Need: A Fast Self-Attention Neural Network Backbone Architecture for the Edge via Double-Condensing Attention Condensers
Revised: Faster Attention Is What You Need: A Fast Self-Attention Neural Network Backbone Architecture for the Edge via Double-Condensing Attention Condensers

Original: Abstract
Revised: Abstract

Original: With the growing adoption of deep learning for on-device TinyML applications,there has been an ever-increasing demand formore efficient neural network backbones optimized for the edge.
Revised: With the growing adoption of deep learning for on-device TinyML applications, there has been an ever-increasing demand for efficient neural network backbones optimized for the edge.

Original: Recently, the introduction of attention condensernetworks have resulted in low-footprint, highly-efficient, self-attention neural networks that strike a strong balance between accuracy and speed.
Revised: Recently, the introduction of attention condenser networks have resulted in low-footprint, highly-efficient, self-attention neural networks that strike a strong balance between accuracy and speed.

Original: In this study,we introduce anew faster attention condenser design called double-condensingattention condensers that enable more condensed feature embedding.
Revised: In this study, we introduce a faster attention condenser design called double-condensing attention condensers that allow for highly condensed feature embeddings.

Original: We furtheremploy a machine-driven design exploration strategy that imposes best practicesdesign constraints  for greater efficiency and robustness to produce the macro-microarchitecture constructs of the backbone.
Revised: We further employ a machinedriven design exploration strategy that imposes design constraints based on best practices for greater efficiency and robustness to produce the macro-micro architecture constructs of the backbone.

Original: The resulting backbone (which we name
Revised: The resulting backbone (which we name

Original: AttendNeXt ) achieves significantly higher inference throughput on an embedded
Revised: AttendNeXt ) achieves significantly higher inference throughput on an embedded

Original: ARM processor when compared to several other state-of-the-art efficient backbones( > 10 × faster than FB-Net C at higher accuracy and speed and > 10 × faster than
Revised: ARM processor when compared to several other state-of-the-art efficient backbones ( > 10 × faster than FB-Net C at higher accuracy and speed and > 10 × faster than

Original: MobileOne-S1 at smaller size) while having a small model size ( > 1 . 37 × smallerthan MobileNetv3-L at higher accuracy and speed) and strong accuracy (1.1%higher top-1 accuracy than MobileViT XS on ImageNet at higher speed).
Revised: MobileOne-S1 at smaller size) while having a small model size ( > 1 . 37 × smaller than MobileNetv3-L at higher accuracy and speed) and strong accuracy (1.1% higher top-1 accuracy than MobileViT XS on ImageNet at higher speed).

Original: Thesepromising results demonstrate that exploring different efficient architecture designsand self-attention mechanisms can lead to interesting new building blocks for
Revised: These promising results demonstrate that exploring different efficient architecture designs and self-attention mechanisms can lead to interesting new building blocks for

Original: TinyML applications.
Revised: TinyML applications.

Original:  Introduction
Revised:  Introduction

Original: Over the past decade, Deep Learning has made tremendous strides in achieving or even exceedinghuman-level performance in image perception tasks such as object segmentation, detection andclassification.
Revised: Over the past decade, Deep Learning has made tremendous strides in achieving or even exceeding human-level performance in image perception tasks such as object segmentation, detection and classification.

Original: Many key emerging technologies such as autonomous driving, automated industrialinspection and augmented/virtual reality applications rely on the aforementioned perception tasks towork effectively.
Revised: Many key emerging technologies such as autonomous driving, automated industrial inspection and augmented/virtual reality applications rely on the aforementioned perception tasks to work effectively.

Original: Most real world applications typically require on-device deployment of deep neuralnetworks (DNNs) for the purposes of real-time inference, privacy and security.
Revised: Most real world applications typically require on-device deployment of deep neural networks (DNNs) for the purposes of real-time inference, privacy and security.

Original: However, DNNstypically require significant on-board computational resources which makes them challenging todeploy on edge devices.
Revised: However, DNNs typically require significant on-board computational resources which makes them challenging to deploy on edge devices.

Original: Given the potential of Deep Learning on the edge, there has been significanteffort in recent years on developing highly efficient DNNs for resource constrained devices [1–5],with different strategies introduced to find a strong balance between efficiency and accuracy.
Revised: Given the potential of Deep Learning on the edge, there has been significant effort in recent years on developing highly efficient DNNs for resource constrained devices [1–5], with different strategies introduced to find a strong balance between efficiency and accuracy.

Original: Forexample, MobileNet [4] and FB-Net [2] architectures leveraged bottleneck inverted residuals blocks to achieve efficient networks.
Revised: For example, MobileNet [4] and FB-Net [2] architectures leveraged bottleneck inverted residuals blocks to achieve efficient networks.

Original: In another example, MobileOne [5] architectures leveraged network reparameterization to eliminate branches within the DNN architecture to achieve low latency inference.
Revised: In another example, MobileOne [5] architectures leveraged network reparameterization to eliminate branches within the DNN architecture to achieve low latency inference.

Original: However, much of the aforementioned methods emphasized identifying efficient architectural patternsto discover optimal DNNs.
Revised: However, much of the aforementioned methods emphasized identifying efficient architectural patterns to discover optimal DNNs.

Original: Very recently, given the success in attention mechanisms in deep learning for improving a neuralnetwork’s focus on relevant stimuli while attenuating irrelevant stimuli, there has been growinginterest in exploring the utility of self-attention mechanisms for constructing efficient DNNs.
Revised: Very recently, given the success in attention mechanisms in deep learning for improving a neural network’s focus on relevant stimuli while attenuating irrelevant stimuli, there has been growing interest in exploring the utility of self-attention mechanisms for constructing efficient DNNs.

Original: Onesuch effort yielded MobileViT [1] architectures which leveraged self-attention within a visiontransformer architectural design to achieve highly efficient DNNs.
Revised: One such effort yielded MobileViT [1] architectures which leveraged self-attention within a vision transformer architectural design to achieve highly efficient DNNs.

Original: Another recent effort yielded theconcept of attention condensers [6, 7], a novel self-attention mechanism that learns and produces acondensed embedding characterizing joint local cross-channel activation relationships.
Revised: Another recent effort yielded the concept of attention condensers [6, 7], a novel self-attention mechanism that learns and produces a condensed embedding characterizing joint local cross-channel activation relationships.

Original: The use ofattention condensers has led to small footprint, highly efficient self-attention DNNs called attentioncondenser networks that strike a strong balance between accuracy and latency [8, 7, 9, 10].
Revised: The use of attention condensers has led to small footprint, highly efficient self-attention DNNs called attention condenser networks that strike a strong balance between accuracy and latency [8, 7, 9, 10].

Original: Assuch, the incorporation of self-attention for improving the balance of efficiency and accuracy in deepneural network architecture design holds tremendous promise but is only in its infancy of scientificexploration.
Revised: As such, the incorporation of self-attention for improving the balance of efficiency and accuracy in deep neural network architecture design holds tremendous promise but is only in its infancy of scientific exploration.

Original: Motivated to further improve the efficiency of attention condenser networks, in this study we introduce a double-condensing attention condenser (DC-AC) self-attention mechanism, an enhancementof the attention condenser mechanism proposed in [6].
Revised: Motivated to further improve the efficiency of attention condenser networks, in this study we introduce a double-condensing attention condenser (DC-AC) self-attention mechanism, an enhancement of the attention condenser mechanism proposed in [6].

Original: The key enhancement to the original attention condenser design is the introduction of an additional condensation mechanism on the inputfeatures, which enables more condensed feature embedding to be learned for improved balancebetween efficiency and representational performance.
Revised: The key enhancement to the original attention condenser design is the introduction of an additional condensation mechanism on the input features, which enables more condensed feature embedding to be learned for improved balance between efficiency and representational performance.

Original: In addition, we leverage a machine-drivendesign exploration strategy [11] that imposes best practices design constraints for greater efficiencyand robustness to produce macro-micro architecture constructs that leverage the proposed DC-ACmechanism, resulting in a fast backbone architecture we name AttendNeXt .
Revised: In addition, we leverage a machine-driven design exploration strategy [11] that imposes best practices design constraints for greater efficiency and robustness to produce macro-micro architecture constructs that leverage the proposed DC-AC mechanism, resulting in a fast backbone architecture we name AttendNeXt .

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: Section 2 details the machine-driven design exploration strategyused to create AttendNeXt, the architecture of the double-condensing attention condenser design, andthe network architecture of AttendNeXt.
Revised: Section 2 details the machine-driven design exploration strategy used to create AttendNeXt, the architecture of the double-condensing attention condenser design, and the network architecture of AttendNeXt.

Original: Section 3 presents the experimental setup of the study alongwith the experimental results and discussion comparing AttendNeXt with state-of-the-art efficientbackbones.
Revised: Section 3 presents the experimental setup of the study along with the experimental results and discussion comparing AttendNeXt with state-of-the-art efficient backbones.

Original:  Methods 
Revised:  Methods 2.1 Double-condensing attention condensers

Original: 2.1 Double-condensing attention condensers

Original: In this study, we introduce a double-condensing attention condenser (DC-AC) self-attention mechanism, an enhancement of the attention condenser mechanism first introduced in [6] for incorporatingefficient selective attention within a network architecture towards relevant stimuli.
Revised: In this study, we introduce a double-condensing attention condenser (DC-AC) self-attention mechanism, an enhancement of the attention condenser mechanism first introduced in [6] for incorporating efficient selective attention within a network architecture towards relevant stimuli.

Original: The architectural design of a DC-AC self-attention mechanism is shown in Figure 1, and can be described asfollows.
Revised: The architectural design of a DC-AC self-attention mechanism is shown in Figure 1, and can be described as follows.

Original: Fundamentally, DC-AC self-attention mechanisms are comprised of: 1) condenser layers, 2)embedding layers, and 3) expansion layers.
Revised: Fundamentally, DC-AC self-attention mechanisms are comprised of: 1) condenser layers, 2) embedding layers, and 3) expansion layers.

Original: More specifically, the condensation layers are responsible for condensing the input V such thatdimensionality is reduced in a manner that places greater emphasis on activations in close proximity tostrong activations.
Revised: More specifically, the condensation layers are responsible for condensing the input V such that dimensionality is reduced in a manner that places greater emphasis on activations in close proximity to strong activations.

Original: This condensation allows for significantly reduced architectural and computationalcomplexity while encouraging the characterization of strongly activated locations.
Revised: This condensation allows for significantly reduced architectural and computational complexity while encouraging the characterization of strongly activated locations.

Original: The embeddinglayers are responsible for learning condensed embeddings characterizing joint local and crosschannel activation relationships.
Revised: The embedding layers are responsible for learning condensed embeddings characterizing joint local and crosschannel activation relationships.

Original: This enables a rich understanding and characterization of what isimportant to focus on in the data while preserving lower architectural and computational complexity.
Revised: This enables a rich understanding and characterization of what is important to focus on in the data while preserving lower architectural and computational complexity.

Original: The expansion layers are responsible for projecting the condensed embeddings to the appropriatedimensionality.
Revised: The expansion layers are responsible for projecting the condensed embeddings to the appropriate dimensionality.

Original: The resulting self-attention values A and feature embeddings B are then leveraged toperform selective attention V = F ( A, B ) , which effectively leads to attenuation of irrelevant stimuliand direct attention to relevant stimuli.
Revised: The resulting self-attention values A and feature embeddings B are then leveraged to perform selective attention V = F ( A, B ) , which effectively leads to attenuation of irrelevant stimuli and direct attention to relevant stimuli.

Revised: The key enhancement to the original attention condenser design is the introduction of an additional condensation mechanism on the input features V (see top feature branch with output B ), resulting in greater symmetry in condensation on both the feature branch and the attention branch in the DC-AC self-attention mechanism.

Revised: This symmetrical condensation within the self-attention mechanism in effect enables more condensed feature embeddings to be learned for improved balance between network efficiency and representational performance.

Revised: We further streamlined the attention condenser design by removing the scaling conducted during selective attention F , which was found to have limited effect in the new DC-AC design.

Revised: The proposed DC-AC mechanism is leveraged during machine-driven design exploration.

Original: AADS 3x3Conv
Revised: AADS 3x3Conv

Original: I N
Revised: I N

Original: DC-AC
Revised: DC-AC

Original: AvgPool FC Softmax 1x1Conv
Revised: AvgPool FC Softmax 1x1Conv

Original: X XX 2x X XX 2x
Revised: X XX 2x X XX 2x

Original: X XX
Revised: X XX

Original: X XX X XX
Revised: X XX X XX

Original: X XX 101x 3x 

Revised: X XX 101x 3x

Original: X XX 2x X XX 2x
Revised: X XX 2x X XX 2x

Original: X XX 2x X XX X XX 2x
Revised: X XX 2x X XX X XX 2x

Original: X XX 102x 

Revised: X XX 102x

Original: X
Revised: X

Original: X
Revised: X

Original: X X
Revised: X X

Original: X
Revised: X

Original: X 2x 2x
Revised: X 2x 2x

Original: X
Revised: X

Original: X
Revised: X

Original: X X
Revised: X X

Original: X
Revised: X

Original: X 2x 2x
Revised: X 2x 2x

Original: X
Revised: X

Original: X X 
Revised: X X X

Original: X
Revised: X

Revised: X 2x 2x

Original: X
Revised: X

Original: X 2x 2x
Revised: X X X X

Original: X XX
Revised: X X

Original: X XX
Revised: X X

Original: XX X X
Revised: X

Original: X
Revised: X

Original: X 2x 2x
Revised: X 2x 2x

Original: X
Revised: X

Original: X
Revised: X

Original: X X
Revised: X X

Original: X
Revised: X

Original: X      2x 2x                   176  372    ×
Revised: X      2x 2x                   176  372    ×

Original: V
Revised: V

Original: B
Revised: B

Original: V'=F(A,B)
Revised: V'=F(A,B)

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: The key enhancement to the original attention condenser design is the introduction of an additionalcondensation mechanism on the input features V (see top feature branch with output B ), resulting ingreater symmetry in condensation on both the feature branch and the attention branch in the DC-ACself-attention mechanism.

Original: This symmetrical condensation within the self-attention mechanism ineffect enables more condensed feature embeddings to be learned for improved balance betweennetwork efficiency and representational performance.

Original: We further streamlined the attention condenserdesign by removing the scaling conducted during selective attention F , which was found to havelimited effect in the new DC-AC design.

Original: The proposed DC-AC mechanism is leveraged duringmachine-driven design exploration.

Original: 2.2 Machine-driven design exploration
Revised: 2.2 Machine-driven design exploration

Original: In this study, generative synthesis [11] is leveraged to perform machine-driven design explorationto determine the macro-micro architecture design of AttendNeXt.
Revised: In this study, generative synthesis [11] is leveraged to perform machine-driven design exploration to determine the macro-micro architecture design of AttendNeXt.

Original: Generative synthesis is an generative architectural exploration process that discovers highly tailored DNN architectures based onoperational requirements and constraints.
Revised: Generative synthesis is an generative architectural exploration process that discovers highly tailored DNN architectures based on operational requirements and constraints.

Original: The exploration process is formulated as a constrainedoptimization problem
Revised: The exploration process is formulated as a constrained optimization problem

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: [Equation] ) where the goal is to find the optimal generator G ⋆ ( · ) which generates network architectures N thatmaximizes a universal performance function U (e.g., [12]) under a given set of operational constraintsformulated via an indicator function 1 r ( · ) .
Revised: (1) where the goal is to find the optimal generator G ⋆ ( · ) which generates network architectures N that maximizes a universal performance function U (e.g., [12]) under a given set of operational constraints formulated via an indicator function 1 r ( · ) .

Original: The optimization process is performed iteratively via theinterplay between a generator G ⋆ ( · ) , which is tasked with generating network architectures N and aninquisitor I responsible for assessing the performance of the G via its generated architectures N .
Revised: The optimization process is performed iteratively via the interplay between a generator G ⋆ ( · ) , which is tasked with generating network architectures N and an inquisitor I responsible for assessing the performance of the G via its generated architectures N .

Original: We impose four key best practices design constraints via 1 r ( · ) to identify macro- and micro- architecture designs for AttendNeXt that exhibit the desired balance between accuracy, architecturalcomplexity, and computational complexity to produce a compact, low footprint, high performanceneural network tailored for edge scenarios.
Revised: We impose four key best practices design constraints via 1 r ( · ) to identify macro- and micro- architecture designs for AttendNeXt that exhibit the desired balance between accuracy, architectural complexity, and computational complexity to produce a compact, low footprint, high performance neural network tailored for edge scenarios.

Original: These four constraints include:
Revised: These four constraints include:

Original: Encouraging columnar architecture designs with parallel columns to significantly reduce[Figure] architectural and computational complexity.[Figure]
Revised: Encouraging columnar architecture designs with parallel columns to significantly reduce architectural and computational complexity.

Original: Restricting the use of point-wise strided convolutions to mitigate the considerable informa- [Figure] tion loss seen in many residual network designs [13, 14].[Figure]
Revised: Restricting the use of point-wise strided convolutions to mitigate the considerable information loss seen in many residual network designs [13, 14].

Original: Encouraging the use of anti-aliased downsampling (AADS) [15] to improve network stability[Figure] and robustness.[Figure]
Revised: Encouraging the use of anti-aliased downsampling (AADS) [15] to improve network stability and robustness.

Original: Enforcing top-1 accuracy on ImageNet for a validation set to be

Original: above

Original: 8% to

Original:  ensure AttendNeXt performs at least as well as OFA-62, a compact architecture created via OnceFor-All ( OFA) a state-of-the-art 
Revised: Enforcing top-1 accuracy on ImageNet for a validation set to be above 75.8% to ensure AttendNeXt performs at least as well as OFA-62, a compact architecture created via OnceFor-All (OFA) [3], a state-of-the-art neural architecture search method.

Original: neural

Original: architecture search method.

Original: 2.3 Network architecture
Revised: 2.3 Network architecture

Original: Figure 1 demonstrates the AttendNeXt architecture designed via machine-driven exploration.
Revised: Figure 1 demonstrates the AttendNeXt architecture designed via machine-driven exploration.

Original: Anumber of observations can be made from Figure 1.
Revised: A number of observations can be made from Figure 1.

Original: First, it possesses a heterogeneous columnardesign with different degrees of columnar interactions at different stages, thus striking a balancebetween accuracy and efficiency by learning representative yet disentangled embeddings.
Revised: First, it possesses a heterogeneous columnar design with different degrees of columnar interactions at different stages, thus striking a balance between accuracy and efficiency by learning representative yet disentangled embeddings.

Original: Secondly,by having more independent columns in the earlier stages and more columnar interactions at laterstages, independent feature learning is amplified for lower levels of abstraction while more complexfeature learning for higher levels of abstraction are catered for through the increased interactionsamongst the columns.
Revised: Secondly, by having more independent columns in the earlier stages and more columnar interactions at later stages, independent feature learning is amplified for lower levels of abstraction while more complex feature learning for higher levels of abstraction are catered for through the increased interactions amongst the columns.

Original: Thirdly, the widespread presence of AADS throughout the network architectureimproves robustness and stability by making it more shift-invariant [15].
Revised: Thirdly, the widespread presence of AADS throughout the network architecture improves robustness and stability by making it more shift-invariant [15].

Original: Finally, we observe that DCAC modules at different stages of the architectures increase efficiency of selective attention.
Revised: Finally, we observe that DCAC modules at different stages of the architectures increase efficiency of selective attention.

Original: Efficientselective attention results in improved representational efficiency through the use of condensedcharacterizations of joint spatial-channel activation relationships.
Revised: Efficient selective attention results in improved representational efficiency through the use of condensed characterizations of joint spatial-channel activation relationships.

Original:  Results
Revised:  Results

Original: The efficacy of the proposed AttendNeXt architecture is evaluated on ImageNet dataset and iscompared with the state-of-the-art efficient architectures (OFA-62 created via Once-For-All (OFA) [3],
Revised: The efficacy of the proposed AttendNeXt architecture is evaluated on ImageNet dataset and is compared with the state-of-the-art efficient architectures (OFA-62 created via Once-For-All (OFA) [3],

Original: FB-Net C [2], MobileViT XS [1], MobileOne-S1 [5], and MobileNetv3-L [4]) across three metrics:1) Top-1 accuracy, 2) model size, and 3) relative inference throughput (with FB-Net C as baseline) onan embedded, low-power ARM Cortex A72 processor often used for edge computing applications.
Revised: FB-Net C [2], MobileViT XS [1], MobileOne-S1 [5], and MobileNetv3-L [4]) across three metrics: 1) Top-1 accuracy, 2) model size, and 3) relative inference throughput (with FB-Net C as baseline) on an embedded, low-power ARM Cortex A72 processor often used for edge computing applications.

Original: Figure 2 shows the proposed AttendNeXt architecture achieved 75.8% top-1 accuracy, outperforming
Revised: Figure 2 shows the proposed AttendNeXt architecture achieved 75.8% top-1 accuracy, outperforming

Original: MobileViT-XS and FB-Net C by ∼ 1%.
Revised: MobileViT-XS and FB-Net C by ∼ 1%.

Original: While AttendNeXt achieves slightly better top-1 accuracy than
Revised: While AttendNeXt achieves slightly better top-1 accuracy than

Original: MobileNetV3-L and OFA-62, it significantly outperforms both architectures in inference throughputon ARM Cortex A72.
Revised: MobileNetV3-L and OFA-62, it significantly outperforms both architectures in inference throughput on ARM Cortex A72.

Original: Figure 3(a) shows that AttendNeXt is > 10 × faster than FB-Net C and
Revised: Figure 3(a) shows that AttendNeXt is > 10 × faster than FB-Net C and

Original: MobileOne-S1, > 6 × faster than OFA-62 and MobileNetv3-L, and ∼ 4 × faster than MobileViT XS.
Revised: MobileOne-S1, > 6 × faster than OFA-62 and MobileNetv3-L, and ∼ 4 × faster than MobileViT XS.

Original: In terms of model size, Figure 3(b) shows that AttendNeXt is significantly smaller than OFA-62,
Revised: In terms of model size, Figure 3(b) shows that AttendNeXt is significantly smaller than OFA-62,

Original: MobileNetv3-L, FB-Net C, and MobileOne-S1 (e.g., AttendNeXt is > 1 . 

Revised: MobileNetv3-L, FB-Net C, and MobileOne-S1 (e.g., AttendNeXt is > 1 .

Original: × smaller than OFA62).
Revised: × smaller than OFA62).

Original: While larger than MobileViT-XS, AttendNeXt is significantly faster on ARM Cortex A72 andsignificantly higher top-1 accuracy.
Revised: While larger than MobileViT-XS, AttendNeXt is significantly faster on ARM Cortex A72 and significantly higher top-1 accuracy.

Original: These results demonstrate that AttendNeXt possesses a strong balance between accuracy, architecturalcomplexity, and computational complexity, making such an architecture well-suited for TinyMLapplications on the edge.
Revised: These results demonstrate that AttendNeXt possesses a strong balance between accuracy, architectural complexity, and computational complexity, making such an architecture well-suited for TinyML applications on the edge.

Original: Furthermore, it illustrates that efficient selective attention can enable astrong balance between network efficiency and representational power.
Revised: Furthermore, it illustrates that efficient selective attention can enable a strong balance between network efficiency and representational power.

Original: These promising results alsodemonstrate that exploring different efficient architecture designs and self-attention mechanisms canlead to interesting new building blocks for TinyML applications.
Revised: These promising results also demonstrate that exploring different efficient architecture designs and self-attention mechanisms can lead to interesting new building blocks for TinyML applications.

Original:  Broader Impact
Revised:  Broader Impact

Original: The interest in TinyML (tiny machine learning) has grown significantly in recent years as one of thekey driving factors towards widespread adoption of machine learning in industry and society.
Revised: The interest in TinyML (tiny machine learning) has grown significantly in recent years as one of the key driving factors towards widespread adoption of machine learning in industry and society.

Original: TinyMLis crucial for enabling real-time decision-making and predictive analytics on low-cost, low-powerembedded devices, and holds tremendous potential that is currently being realized across industriesranging from manufacturing to energy to automotive to aerospace to healthcare.
Revised: TinyML is crucial for enabling real-time decision-making and predictive analytics on low-cost, low-power embedded devices, and holds tremendous potential that is currently being realized across industries ranging from manufacturing to energy to automotive to aerospace to healthcare.

Original: The proliferation of
Revised: The proliferation of

Original: TinyML can have considerable important socioeconomic implications that needs to be consideredgiven its ability to enable machine learning across industries and applications.
Revised: TinyML can have considerable important socioeconomic implications that needs to be considered given its ability to enable machine learning across industries and applications.

Original: Furthermore, theinterest in attention mechanisms within deep learning has greatly increased in recent years, andits proliferation also has important implications given its increasing adoption within real-worldapplications.
Revised: Furthermore, the interest in attention mechanisms within deep learning has greatly increased in recent years and its proliferation also has important implications given its increasing adoption within real-world applications.

Original: The hope with explorations such as this study that introduces new building blocks suchas double-condensing attention condensers is that it would provide new insights in advancing effortsin TinyML for greater adoption of machine learning towards ubiquity, as well as shed new insightsinto selective attention within a neural network context.
Revised: The hope with explorations such as this study that introduces new building blocks such as double-condensing attention condensers is that it would provide new insights in advancing efforts in TinyML for greater adoption of machine learning towards ubiquity as well as shed new insights into selective attention within a neural network context