Focus Your Attention (with Adaptive IIR Filters) (2023)

Abstract: We present a new layer in which dynamic (i.e., input-dependent) Infinite Impulse Response (IIR) filters of order two are used to process the input sequence prior to applying conventional attention. The input is split into chunks, and the coefficients of these filters are determined based on previous chunks to maintain causality. Despite their relatively low order, the causal adaptive filters are shown to focus attention on the relevant sequence elements. The new layer is grounded in control theory, and is shown to generalize diagonal state-space layers. The layer performs on-par with state-of-the-art networks, with a fraction of their parameters and with time complexity that is sub-quadratic with input size. The obtained layer is favorable to layers such as Heyna, GPT2, and Mega, both with respect to the number of parameters and the obtained level of performance on multiple long-range sequence problems.

## introduction designing sequence models that capture short-and long-term dependencies is a central goal of sequence modeling. besides performance, computational complexity also plays a part when dealing with long sequences. although transformers @xcite excel at tasks that involve short-range dependencies, their performance on data with long-range dependencies can be poor. for example, regardless of the high time complexity, on the long range arena benchmark (lra) @xcite transformers perform poorly compared to other sequence models. another approach that has emerged to address long-range data processing is the utilization of regularized implicit global (long) convolutions. in this technique, convolutions are employed along the sequence dimension, enabling convolutions with a global receptive field. initially, this approach was implemented through state-space layers @xcite , which introduced a recurrent layer that could be efficiently computed via a global convolution. recent research has explored alternative variations of global convolutions, including implicit @xcite and regularized parameterization @xcite . these methods have demonstrated improved performance on tasks involving long-range dependencies and with sub-quantitative complexity. they have also shown effectiveness in enhancing long-range transformer capabilities @xcite @xcite . however, it remains uncertain whether these models can scale up or function similarly to transformers across a diverse range of tasks @xcite . this work strives to efficiently integrate convolution-based sequence models and transformers, to provide a model that is capable of handling both short and long dependencies. the attempt to combine these components was first presented by @xcite , who used a simple global convolution before each transformer block. this convolution is parameterized by the exponential moving average (ema) recurrent rule and can be seen as an iir filter. in this work, instead of using first-order iir filters, we introduce learnable adaptive iir filters, which allow us to propose focus, a layer that combines local attention and a novel type of regularized global convolution grounded on a hypernetwork that produces adaptive iir filters. our main contribution is the focus layer, which has several unique properties: (i) we are the first to use data-dependent global filters, which are implemented by a global hyper-network mechanism that focuses local attention. (ii) in contrast to other works in the domain that employ fir filters, it relies on iir filters. (iii). we present an efficient and stable computation of those iir filters. (iv) theoretically, our layer is grounded in the theory of control systems, similar to state-space layers, which are built on the state-space model (ssm) of control theory. furthermore, in sec. 4 we show that iir filters are a generalization of ssms and diagonal-linear rnns, which have recently been recognized as remarkable long-range learning architectures @xcite @xcite @xcite . by drawing upon the extensive research conducted on iir filters, our findings can provide additional insights into the effectiveness, stability, expressiveness, and initialization of those layers. the state-space representation of an iir filter is a convenient way to represent the filter's dynamics and to implement it in software. it consists of three parts: the state vector, the state transition matrix, and the output matrix. the state vector contains the filter's internal state variables, the state transition matrix describes how the state vector changes over time, and the output matrix describes how the output signal is computed from the state vector. such representation are described in @xcite learnable iir filters since iir filters are computationally efficient, yet expressive, it is natural to design iir filters with deep learning. @xcite proposes an approach to using traditional digital iir filter structures inside deeplearning networks trained using backpropagation. the authors establish the link between such structures and recurrent neural networks and present three different differentiable iir filter topologies. they compare the proposed topologies against each other and an established baseline and show that the proposed topologies can achieve better performance in some cases. additionally, the authors present a simple wiener-hammerstein model, using differentiable iirs as its filtering component, and train it on a guitar signal. global convolutions the global convolution, also known as a long convolution, is a layer that applies scalar convolutions along the sequence dimension, enabling the handling of unrestricted 1-d sequences. empirically, these layers have shown strong performance in tasks involving long-range dependencies, particularly in domains such as nlp @xcite @xcite , audio @xcite , speech @xcite , video @xcite , time-series analysis @xcite and more. moreover, they exhibit computational efficiency as their cost is sub-quadratic. however, to achieve sota results, appropriate regularization is necessary. the approach of @xcite @xcite incorporates a parameterization that inherently regularizes the kernel and decoupling sequence length from parameter count. @xcite utilizes an implicit parameterization learned by ffns operating on positional encodings, while @xcite explicitly regularizes the convolution kernels using squash or smooth operators. ## experiments below we present experimental results for the proposed focus layer. in addition to our full method, we introduce an ablation to evaluate the importance of adaptive filtering, in which instead of the hypernetwork h, the iir filters are conventional learned parameters. this ablation is denoted by "focus-h". ## conclusions attention models are extremely powerful for modeling sequences, as demonstrated by the seminal work of @xcite . indeed, transformers have revolutionized the way deep learning is practiced, leading to unprecedented performance across almost all studied ai domains. however, transformers have quadratic complexity in the sequence length, which can impact their efficiency, and often struggle to perform optimally in tasks that involve long-range dependencies @xcite . in this work, we present a dynamic filtering approach that enables us to subsequently employ attention within much shallower architectures. as our ablation shows, the dynamic nature of these filters is crucial to the success of the layer. similarly crucial is the use of iir filters, and we analyze the regime in which these are stable.